Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:15.485250 22551 test_util.cc:268] Using random seed: -463914479
I20240501 13:59:15.496943 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:15.497026 22551 ts_itest-base.cc:113] --------------
I20240501 13:59:15.497069 22551 ts_itest-base.cc:114] 3 tablet servers
I20240501 13:59:15.497159 22551 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:15.497224 22551 ts_itest-base.cc:116] --------------
2024-05-01T13:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:15Z Disabled control of system clock
I20240501 13:59:15.522927 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:35229
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:35229
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 13:59:15.852469 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.852819 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.852906 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.860594 22563 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:15.860762 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:15.860824 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.860882 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:15.860929 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:15.870363 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:35229
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.871765 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.873658 22563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.885310 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.885802 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.886410 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.887207 22563 server_base.cc:1034] running on GCE node
I20240501 13:59:15.888038 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.889710 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.890931 22563 hybrid_clock.cc:648] HybridClock initialized: now 1714571955890894 us; error 35 us; skew 500 ppm
I20240501 13:59:15.891320 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.897286 22563 webserver.cc:460] Webserver started at http://127.22.5.254:33739/ using document root <none> and password file <none>
I20240501 13:59:15.898113 22563 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.898258 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.898576 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.901549 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3134584099bc4b948beba33a75bf147d"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-5jzj"
I20240501 13:59:15.902362 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3134584099bc4b948beba33a75bf147d"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-5jzj"
I20240501 13:59:15.908041 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:15.912030 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.913480 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:15.913661 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3134584099bc4b948beba33a75bf147d"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-5jzj"
I20240501 13:59:15.913874 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.936146 22563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.937088 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.937417 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.957033 22563 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:35229
I20240501 13:59:15.957051 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:35229 every 8 connection(s)
I20240501 13:59:15.958521 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:15.961098 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 22563
I20240501 13:59:15.961393 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:15.962604 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.972607 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d: Bootstrap starting.
I20240501 13:59:15.975680 22629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.976739 22629 log.cc:824] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:15.980257 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d: No bootstrap required, opened a new log
I20240501 13:59:15.988395 22629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } }
I20240501 13:59:15.988862 22629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.988986 22629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3134584099bc4b948beba33a75bf147d, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.989789 22629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } }
I20240501 13:59:15.990065 22629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:15.990195 22629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:15.990367 22629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.992944 22629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } }
I20240501 13:59:15.993502 22629 leader_election.cc:302] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3134584099bc4b948beba33a75bf147d; no voters:
I20240501 13:59:15.994076 22629 leader_election.cc:288] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:15.994210 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:15.994835 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [term 1 LEADER]: Becoming Leader. State: Replica: 3134584099bc4b948beba33a75bf147d, State: Running, Role: LEADER
I20240501 13:59:15.995388 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } }
I20240501 13:59:15.996074 22629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:15.998373 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3134584099bc4b948beba33a75bf147d. Latest consensus state: current_term: 1 leader_uuid: "3134584099bc4b948beba33a75bf147d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } } }
I20240501 13:59:15.998764 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:16.000025 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3134584099bc4b948beba33a75bf147d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134584099bc4b948beba33a75bf147d" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 35229 } } }
I20240501 13:59:16.000378 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:16.002512 22639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:16.008538 22639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:16.017768 22639 catalog_manager.cc:1347] Generated new cluster ID: 8c994eb34cd2450b863432ed0c074668
I20240501 13:59:16.017907 22639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:16.040272 22639 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:16.041555 22639 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:16.048610 22639 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3134584099bc4b948beba33a75bf147d: Generated new TSK 0
I20240501 13:59:16.049759 22639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:16.067904 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--builtin_ntp_servers=127.22.5.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:16.434528 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:16.434844 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:16.434944 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:16.442667 22653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:16.442802 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:16.442930 22653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 13:59:16.452067 22653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:16.453485 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:16.455232 22653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:16.467614 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.468159 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.468816 22661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:16.469442 22653 server_base.cc:1034] running on GCE node
I20240501 13:59:16.470090 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:16.471076 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:16.472282 22653 hybrid_clock.cc:648] HybridClock initialized: now 1714571956472235 us; error 46 us; skew 500 ppm
I20240501 13:59:16.472607 22653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:16.474983 22653 webserver.cc:460] Webserver started at http://127.22.5.193:33605/ using document root <none> and password file <none>
I20240501 13:59:16.475761 22653 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:16.475878 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:16.476195 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.478945 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.479743 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.485219 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 13:59:16.488905 22667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.490777 22653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:16.490957 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.491180 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.524138 22653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.525076 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.525416 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.526443 22653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.527961 22653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.528060 22653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.528183 22653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.528260 22653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.560667 22653 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:38975
I20240501 13:59:16.560724 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:38975 every 8 connection(s)
I20240501 13:59:16.562075 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:16.569999 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 22653
I20240501 13:59:16.570197 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:16.573789 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--builtin_ntp_servers=127.22.5.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:16.575285 22780 heartbeater.cc:347] Connected to a master server at 127.22.5.254:35229
I20240501 13:59:16.575603 22780 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:16.576437 22780 heartbeater.cc:511] Master 127.22.5.254:35229 requested a full tablet report, sending...
I20240501 13:59:16.578622 22594 ts_manager.cc:194] Registered new tserver with Master: 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975)
I20240501 13:59:16.580143 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:59511
W20240501 13:59:16.904508 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:16.904822 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:16.904971 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:16.912426 22784 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:16.912585 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:16.912755 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 13:59:16.922163 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:16.923569 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:16.925393 22784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:16.937822 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.938205 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.938686 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:16.939545 22784 server_base.cc:1034] running on GCE node
I20240501 13:59:16.940227 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:16.941298 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:16.942545 22784 hybrid_clock.cc:648] HybridClock initialized: now 1714571956942507 us; error 52 us; skew 500 ppm
I20240501 13:59:16.942876 22784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:16.945278 22784 webserver.cc:460] Webserver started at http://127.22.5.194:35509/ using document root <none> and password file <none>
I20240501 13:59:16.946051 22784 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:16.946170 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:16.946485 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.949105 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "49e10d94bfc743e69509789884c0e3f4"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.949930 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "49e10d94bfc743e69509789884c0e3f4"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.956238 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:16.961089 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.962824 22784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:16.962994 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "49e10d94bfc743e69509789884c0e3f4"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-5jzj"
I20240501 13:59:16.963232 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.986416 22784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.987274 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.987540 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.988379 22784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.989898 22784 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.989984 22784 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.990068 22784 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.990120 22784 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.022069 22784 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:39259
I20240501 13:59:17.022181 22910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:39259 every 8 connection(s)
I20240501 13:59:17.023473 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:17.032681 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 22784
I20240501 13:59:17.032886 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:17.036444 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--builtin_ntp_servers=127.22.5.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:17.037180 22911 heartbeater.cc:347] Connected to a master server at 127.22.5.254:35229
I20240501 13:59:17.037477 22911 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:17.038275 22911 heartbeater.cc:511] Master 127.22.5.254:35229 requested a full tablet report, sending...
I20240501 13:59:17.039616 22594 ts_manager.cc:194] Registered new tserver with Master: 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:17.040367 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:41535
W20240501 13:59:17.366819 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:17.367132 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:17.367266 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:17.374912 22914 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:17.375042 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:17.375219 22914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 13:59:17.384383 22914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:35229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:17.385836 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:17.387692 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:17.400707 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.401190 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.401665 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:17.402561 22914 server_base.cc:1034] running on GCE node
I20240501 13:59:17.403229 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:17.404274 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:17.405508 22914 hybrid_clock.cc:648] HybridClock initialized: now 1714571957405456 us; error 45 us; skew 500 ppm
I20240501 13:59:17.405865 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:17.408246 22914 webserver.cc:460] Webserver started at http://127.22.5.195:40813/ using document root <none> and password file <none>
I20240501 13:59:17.409029 22914 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:17.409176 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:17.409516 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:17.412220 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-5jzj"
I20240501 13:59:17.413026 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-5jzj"
I20240501 13:59:17.418502 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 13:59:17.422308 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.423975 22914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:17.424153 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-5jzj"
I20240501 13:59:17.424379 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:17.445351 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:17.446334 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:17.446627 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:17.447633 22914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:17.449203 22914 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:17.449311 22914 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.449432 22914 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:17.449507 22914 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.481498 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:40325
I20240501 13:59:17.481544 23040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:40325 every 8 connection(s)
I20240501 13:59:17.482952 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:17.484149 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 22914
I20240501 13:59:17.484359 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:17.496997 23041 heartbeater.cc:347] Connected to a master server at 127.22.5.254:35229
I20240501 13:59:17.497416 23041 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:17.498314 23041 heartbeater.cc:511] Master 127.22.5.254:35229 requested a full tablet report, sending...
I20240501 13:59:17.499856 22594 ts_manager.cc:194] Registered new tserver with Master: 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:17.500734 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:45697
I20240501 13:59:17.501614 22551 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 13:59:17.523240 22551 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 13:59:17.583125 22780 heartbeater.cc:503] Master 127.22.5.254:35229 was elected leader, sending a full tablet report...
I20240501 13:59:18.043336 22911 heartbeater.cc:503] Master 127.22.5.254:35229 was elected leader, sending a full tablet report...
I20240501 13:59:18.503955 23041 heartbeater.cc:503] Master 127.22.5.254:35229 was elected leader, sending a full tablet report...
I20240501 13:59:21.277557 22593 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53892:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240501 13:59:21.284009 22593 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:59:21.318770 22973 tablet_service.cc:1467] Processing CreateTablet for tablet b09d9430f6aa4507a4f16d7796c2965b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.320500 22976 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4a9b798b5e4a308a59bd8d45811c5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.321477 22976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4a9b798b5e4a308a59bd8d45811c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.323464 22968 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.323841 22971 tablet_service.cc:1467] Processing CreateTablet for tablet 67dac4f984e54979b432ebde9a9b5d0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.324061 22968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.324472 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dac4f984e54979b432ebde9a9b5d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.325330 22972 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0da15dc67d47a48a5ac69927258738 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.324910 22965 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc17872dfe046919f1540dedf2f806c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.325659 22962 tablet_service.cc:1467] Processing CreateTablet for tablet 73c2dcdd384e484d8f95f47109f19e69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.325888 22972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0da15dc67d47a48a5ac69927258738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.326237 22965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc17872dfe046919f1540dedf2f806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.326470 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c2dcdd384e484d8f95f47109f19e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.326751 22963 tablet_service.cc:1467] Processing CreateTablet for tablet b8edade9a59f404e903fdc51cb13a708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.327332 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8edade9a59f404e903fdc51cb13a708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.327325 22961 tablet_service.cc:1467] Processing CreateTablet for tablet 5b020e221592443ba62fe794ae418a8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.327719 22969 tablet_service.cc:1467] Processing CreateTablet for tablet 87be1a7f20ab421da40eaa3aed61938c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.328461 22964 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3610a4cab54c2caf976751927b1eb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.328505 22974 tablet_service.cc:1467] Processing CreateTablet for tablet a255d18518e64e789f16f4dd14afcbdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.329310 22970 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5445f1dc614c41ac4bd83d1f249d82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.330294 22975 tablet_service.cc:1467] Processing CreateTablet for tablet a2c272fe58b54fbcb2e4dfda95f46977 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.331506 22967 tablet_service.cc:1467] Processing CreateTablet for tablet 13a435e74d9e4fd5b3a40fd6483cd460 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.336459 22966 tablet_service.cc:1467] Processing CreateTablet for tablet b598b69784ab4492957ea8c8e615db3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.336268 22844 tablet_service.cc:1467] Processing CreateTablet for tablet a255d18518e64e789f16f4dd14afcbdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.336773 22846 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4a9b798b5e4a308a59bd8d45811c5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.337615 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5445f1dc614c41ac4bd83d1f249d82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.337864 22846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4a9b798b5e4a308a59bd8d45811c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.338310 22838 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.339004 22842 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0da15dc67d47a48a5ac69927258738 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.339615 22841 tablet_service.cc:1467] Processing CreateTablet for tablet 67dac4f984e54979b432ebde9a9b5d0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.340037 22839 tablet_service.cc:1467] Processing CreateTablet for tablet 87be1a7f20ab421da40eaa3aed61938c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.335554 22845 tablet_service.cc:1467] Processing CreateTablet for tablet a2c272fe58b54fbcb2e4dfda95f46977 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.336514 22843 tablet_service.cc:1467] Processing CreateTablet for tablet b09d9430f6aa4507a4f16d7796c2965b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.327899 22961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b020e221592443ba62fe794ae418a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.342286 22966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b598b69784ab4492957ea8c8e615db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.342564 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5445f1dc614c41ac4bd83d1f249d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.343760 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87be1a7f20ab421da40eaa3aed61938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.345297 22964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3610a4cab54c2caf976751927b1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.342687 22711 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0da15dc67d47a48a5ac69927258738 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.346926 22713 tablet_service.cc:1467] Processing CreateTablet for tablet a255d18518e64e789f16f4dd14afcbdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.347613 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0da15dc67d47a48a5ac69927258738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.347662 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a255d18518e64e789f16f4dd14afcbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.344506 22706 tablet_service.cc:1467] Processing CreateTablet for tablet 13a435e74d9e4fd5b3a40fd6483cd460 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.349604 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a435e74d9e4fd5b3a40fd6483cd460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.351305 22967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a435e74d9e4fd5b3a40fd6483cd460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.344115 22704 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc17872dfe046919f1540dedf2f806c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.353425 22713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a255d18518e64e789f16f4dd14afcbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.343727 22705 tablet_service.cc:1467] Processing CreateTablet for tablet b598b69784ab4492957ea8c8e615db3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.354032 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b598b69784ab4492957ea8c8e615db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.345010 22712 tablet_service.cc:1467] Processing CreateTablet for tablet b09d9430f6aa4507a4f16d7796c2965b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.357436 22975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c272fe58b54fbcb2e4dfda95f46977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.357558 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09d9430f6aa4507a4f16d7796c2965b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.359997 22704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc17872dfe046919f1540dedf2f806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.343331 22707 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.365656 22707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.331867 22710 tablet_service.cc:1467] Processing CreateTablet for tablet 67dac4f984e54979b432ebde9a9b5d0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.366199 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dac4f984e54979b432ebde9a9b5d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.335629 22835 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc17872dfe046919f1540dedf2f806c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.345625 22715 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4a9b798b5e4a308a59bd8d45811c5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.368376 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8ea9c9db814093a45e2f4d0e4a5e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.368782 22845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c272fe58b54fbcb2e4dfda95f46977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.369190 22842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0da15dc67d47a48a5ac69927258738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.370113 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dac4f984e54979b432ebde9a9b5d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.370337 22839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87be1a7f20ab421da40eaa3aed61938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.370669 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc17872dfe046919f1540dedf2f806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.370754 22836 tablet_service.cc:1467] Processing CreateTablet for tablet b598b69784ab4492957ea8c8e615db3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.336181 22834 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3610a4cab54c2caf976751927b1eb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.374361 22833 tablet_service.cc:1467] Processing CreateTablet for tablet b8edade9a59f404e903fdc51cb13a708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.375592 22843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09d9430f6aa4507a4f16d7796c2965b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.375864 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a255d18518e64e789f16f4dd14afcbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.376245 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5445f1dc614c41ac4bd83d1f249d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.344777 22714 tablet_service.cc:1467] Processing CreateTablet for tablet a2c272fe58b54fbcb2e4dfda95f46977 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.377456 22834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3610a4cab54c2caf976751927b1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.341840 22708 tablet_service.cc:1467] Processing CreateTablet for tablet 87be1a7f20ab421da40eaa3aed61938c (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.378242 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4a9b798b5e4a308a59bd8d45811c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.343055 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5445f1dc614c41ac4bd83d1f249d82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.379040 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5445f1dc614c41ac4bd83d1f249d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.376696 22832 tablet_service.cc:1467] Processing CreateTablet for tablet 73c2dcdd384e484d8f95f47109f19e69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.377593 22831 tablet_service.cc:1467] Processing CreateTablet for tablet 5b020e221592443ba62fe794ae418a8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.375476 22837 tablet_service.cc:1467] Processing CreateTablet for tablet 13a435e74d9e4fd5b3a40fd6483cd460 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.382434 22973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09d9430f6aa4507a4f16d7796c2965b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.384090 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c272fe58b54fbcb2e4dfda95f46977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.384660 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87be1a7f20ab421da40eaa3aed61938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.386410 22836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b598b69784ab4492957ea8c8e615db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.387105 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b020e221592443ba62fe794ae418a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.387943 22832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c2dcdd384e484d8f95f47109f19e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.388684 22712 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3610a4cab54c2caf976751927b1eb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.389153 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3610a4cab54c2caf976751927b1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.393239 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8edade9a59f404e903fdc51cb13a708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.395535 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a435e74d9e4fd5b3a40fd6483cd460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.418797 23059 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.424180 23059 tablet_bootstrap.cc:654] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.425750 23059 log.cc:824] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.431082 22704 tablet_service.cc:1467] Processing CreateTablet for tablet b8edade9a59f404e903fdc51cb13a708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.374737 22702 tablet_service.cc:1467] Processing CreateTablet for tablet 5b020e221592443ba62fe794ae418a8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 13:59:21.374699 22703 tablet_service.cc:1467] Processing CreateTablet for tablet 73c2dcdd384e484d8f95f47109f19e69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=713c8d1f89ad4ccd97ff6be117ded80e]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 13:59:21.432052 23060 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.432495 22704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8edade9a59f404e903fdc51cb13a708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.432880 22702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b020e221592443ba62fe794ae418a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.433320 22703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c2dcdd384e484d8f95f47109f19e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.435369 23059 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.435673 23059 ts_tablet_manager.cc:1367] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240501 13:59:21.438218 23060 tablet_bootstrap.cc:654] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.439551 23060 log.cc:824] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.445281 23059 raft_consensus.cc:348] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.445943 23059 raft_consensus.cc:374] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.446115 23059 raft_consensus.cc:729] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.447091 23059 consensus_queue.cc:260] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.455556 23059 ts_tablet_manager.cc:1398] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20240501 13:59:21.458303 23059 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.460997 23059 tablet_bootstrap.cc:654] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.463923 23059 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.464151 23059 ts_tablet_manager.cc:1367] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240501 13:59:21.465255 23059 raft_consensus.cc:348] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.465570 23059 raft_consensus.cc:374] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.465709 23059 raft_consensus.cc:729] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.466001 23059 consensus_queue.cc:260] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.467252 23060 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.467607 23060 ts_tablet_manager.cc:1367] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.023s
I20240501 13:59:21.468938 23059 ts_tablet_manager.cc:1398] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:21.469341 23059 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.471663 23059 tablet_bootstrap.cc:654] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.475704 23060 raft_consensus.cc:348] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.476107 23060 raft_consensus.cc:374] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.476217 23060 raft_consensus.cc:729] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.476884 23060 consensus_queue.cc:260] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.480057 23060 ts_tablet_manager.cc:1398] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240501 13:59:21.480526 23060 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.482373 23060 tablet_bootstrap.cc:654] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
W20240501 13:59:21.488050 23042 tablet.cc:2369] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:21.491513 23060 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.491729 23060 ts_tablet_manager.cc:1367] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 13:59:21.492777 23060 raft_consensus.cc:348] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.493007 23060 raft_consensus.cc:374] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.493137 23060 raft_consensus.cc:729] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.493399 23060 consensus_queue.cc:260] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.494434 23065 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.496142 23060 ts_tablet_manager.cc:1398] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:21.497532 23065 tablet_bootstrap.cc:654] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.498548 23065 log.cc:824] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.498942 23060 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.500305 23059 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.500475 23059 ts_tablet_manager.cc:1367] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.009s
I20240501 13:59:21.500703 23060 tablet_bootstrap.cc:654] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.501113 23059 raft_consensus.cc:348] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.501370 23059 raft_consensus.cc:374] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.501451 23059 raft_consensus.cc:729] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.501715 23059 consensus_queue.cc:260] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.501822 23065 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.502166 23065 ts_tablet_manager.cc:1367] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 13:59:21.502370 23059 ts_tablet_manager.cc:1398] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.502699 23059 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.504334 23059 tablet_bootstrap.cc:654] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.509655 23060 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.509863 23060 ts_tablet_manager.cc:1367] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 13:59:21.510599 23060 raft_consensus.cc:348] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.510844 23060 raft_consensus.cc:374] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.510916 23060 raft_consensus.cc:729] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.511197 23060 consensus_queue.cc:260] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.512521 23060 ts_tablet_manager.cc:1398] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.512269 23065 raft_consensus.cc:348] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.512743 23065 raft_consensus.cc:374] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.512837 23060 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.512852 23065 raft_consensus.cc:729] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.513737 23065 consensus_queue.cc:260] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.514494 23060 tablet_bootstrap.cc:654] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.515467 23059 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.515659 23059 ts_tablet_manager.cc:1367] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240501 13:59:21.516402 23059 raft_consensus.cc:348] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.516621 23059 raft_consensus.cc:374] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.516716 23060 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.516728 23059 raft_consensus.cc:729] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.516852 23060 ts_tablet_manager.cc:1367] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:21.516952 23059 consensus_queue.cc:260] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.517568 23059 ts_tablet_manager.cc:1398] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:21.517561 23060 raft_consensus.cc:348] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.517767 23060 raft_consensus.cc:374] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.517853 23060 raft_consensus.cc:729] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.517972 23059 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.518061 23060 consensus_queue.cc:260] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.518786 23060 ts_tablet_manager.cc:1398] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.519057 23060 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.520185 23059 tablet_bootstrap.cc:654] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.520954 23060 tablet_bootstrap.cc:654] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.522617 23065 ts_tablet_manager.cc:1398] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.020s user 0.006s sys 0.010s
I20240501 13:59:21.523075 23065 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.524783 23065 tablet_bootstrap.cc:654] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.527449 23065 tablet_bootstrap.cc:492] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.527590 23065 ts_tablet_manager.cc:1367] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
W20240501 13:59:21.528599 22912 tablet.cc:2369] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:21.528646 23065 raft_consensus.cc:348] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.529217 23065 raft_consensus.cc:374] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.529323 23065 raft_consensus.cc:729] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.529657 23065 consensus_queue.cc:260] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.530244 23065 ts_tablet_manager.cc:1398] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:21.530517 23065 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.532222 23065 tablet_bootstrap.cc:654] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.532569 23060 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.532768 23060 ts_tablet_manager.cc:1367] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 13:59:21.533437 23059 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.533668 23059 ts_tablet_manager.cc:1367] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20240501 13:59:21.533636 23060 raft_consensus.cc:348] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.533861 23060 raft_consensus.cc:374] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.533943 23060 raft_consensus.cc:729] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.534245 23060 consensus_queue.cc:260] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.534878 23060 ts_tablet_manager.cc:1398] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.534752 23059 raft_consensus.cc:348] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.535017 23059 raft_consensus.cc:374] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.535122 23059 raft_consensus.cc:729] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.535251 23060 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.535432 23059 consensus_queue.cc:260] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.535950 23059 ts_tablet_manager.cc:1398] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.536353 23059 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.537047 23060 tablet_bootstrap.cc:654] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.538017 23059 tablet_bootstrap.cc:654] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.541054 23065 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.541260 23065 ts_tablet_manager.cc:1367] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 13:59:21.542133 23059 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.542153 23060 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.542073 23065 raft_consensus.cc:348] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.542304 23059 ts_tablet_manager.cc:1367] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 13:59:21.542325 23065 raft_consensus.cc:374] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.542424 23065 raft_consensus.cc:729] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.542519 23060 ts_tablet_manager.cc:1367] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:59:21.542677 23065 consensus_queue.cc:260] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.543176 23065 ts_tablet_manager.cc:1398] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:21.543098 23059 raft_consensus.cc:348] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.543284 23059 raft_consensus.cc:374] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.543387 23059 raft_consensus.cc:729] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.543485 23065 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.543654 23059 consensus_queue.cc:260] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.543713 23060 raft_consensus.cc:348] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.544018 23060 raft_consensus.cc:374] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.544155 23060 raft_consensus.cc:729] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.544211 23059 ts_tablet_manager.cc:1398] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.544518 23059 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.544502 23060 consensus_queue.cc:260] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.545104 23060 ts_tablet_manager.cc:1398] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:21.545269 23065 tablet_bootstrap.cc:654] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.545436 23067 raft_consensus.cc:482] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.545498 23060 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.545693 23067 raft_consensus.cc:504] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.547161 23060 tablet_bootstrap.cc:654] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.547896 23067 leader_election.cc:288] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.551666 23059 tablet_bootstrap.cc:654] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.560155 23065 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.560321 23065 ts_tablet_manager.cc:1367] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20240501 13:59:21.561085 23065 raft_consensus.cc:348] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.561328 23065 raft_consensus.cc:374] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.561422 23065 raft_consensus.cc:729] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.561650 23065 consensus_queue.cc:260] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.562280 23065 ts_tablet_manager.cc:1398] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.562534 23065 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.564142 23065 tablet_bootstrap.cc:654] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.566988 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
W20240501 13:59:21.569770 22781 tablet.cc:2369] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:21.569914 22669 leader_election.cc:341] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.578397 23059 tablet_bootstrap.cc:492] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.578632 23059 ts_tablet_manager.cc:1367] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.002s
I20240501 13:59:21.578716 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.579721 23059 raft_consensus.cc:348] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.580047 23059 raft_consensus.cc:374] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.580130 23059 raft_consensus.cc:729] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.580499 23059 consensus_queue.cc:260] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
W20240501 13:59:21.580750 22671 leader_election.cc:341] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.580976 22671 leader_election.cc:302] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:21.581454 23067 raft_consensus.cc:2726] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:21.582275 23065 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.582456 23065 ts_tablet_manager.cc:1367] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.010s
I20240501 13:59:21.583195 23065 raft_consensus.cc:348] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.583439 23065 raft_consensus.cc:374] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.583525 23065 raft_consensus.cc:729] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.583809 23065 consensus_queue.cc:260] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.584439 23065 ts_tablet_manager.cc:1398] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.585104 23065 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.585884 23059 ts_tablet_manager.cc:1398] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 13:59:21.587458 23065 tablet_bootstrap.cc:654] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.587388 23063 raft_consensus.cc:482] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.587677 23063 raft_consensus.cc:504] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.590171 23063 leader_election.cc:288] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.596570 23067 raft_consensus.cc:482] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.596832 23067 raft_consensus.cc:504] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.597767 23067 leader_election.cc:288] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.598258 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.598436 23063 raft_consensus.cc:482] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.598608 23063 raft_consensus.cc:504] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
W20240501 13:59:21.598971 22671 leader_election.cc:341] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.599280 23063 leader_election.cc:288] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.600982 23059 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.604151 23065 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.604326 23065 ts_tablet_manager.cc:1367] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20240501 13:59:21.605104 23065 raft_consensus.cc:348] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.605340 23065 raft_consensus.cc:374] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.605439 23065 raft_consensus.cc:729] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.605614 23059 tablet_bootstrap.cc:654] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.605683 23065 consensus_queue.cc:260] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.606334 23065 ts_tablet_manager.cc:1398] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:21.607131 23065 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.608820 23065 tablet_bootstrap.cc:654] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.609409 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.611747 23065 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.611907 23065 ts_tablet_manager.cc:1367] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
W20240501 13:59:21.612087 22669 leader_election.cc:341] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.612254 22669 leader_election.cc:302] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:21.612648 23065 raft_consensus.cc:348] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.612824 23067 raft_consensus.cc:2726] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:21.612862 23065 raft_consensus.cc:374] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.612920 23065 raft_consensus.cc:729] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.613178 23065 consensus_queue.cc:260] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.613834 23065 ts_tablet_manager.cc:1398] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.614183 23065 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.614768 23060 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.615079 23060 ts_tablet_manager.cc:1367] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.070s user 0.026s sys 0.034s
I20240501 13:59:21.615437 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.616322 23065 tablet_bootstrap.cc:654] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.616262 23060 raft_consensus.cc:348] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.616547 23060 raft_consensus.cc:374] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.616641 23060 raft_consensus.cc:729] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.617092 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a435e74d9e4fd5b3a40fd6483cd460" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.617482 22735 raft_consensus.cc:2454] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
W20240501 13:59:21.618463 22929 leader_election.cc:341] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.619000 22929 leader_election.cc:302] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.619320 23060 consensus_queue.cc:260] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.619838 23063 raft_consensus.cc:2781] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.619966 23063 raft_consensus.cc:482] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.620085 23063 raft_consensus.cc:3037] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.620354 23060 ts_tablet_manager.cc:1398] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 13:59:21.622853 23060 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.623188 23063 raft_consensus.cc:504] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.624112 23063 leader_election.cc:288] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.625746 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a435e74d9e4fd5b3a40fd6483cd460" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.626003 22735 raft_consensus.cc:3037] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.626168 23060 tablet_bootstrap.cc:654] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.629262 23065 tablet_bootstrap.cc:492] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.629427 23065 ts_tablet_manager.cc:1367] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240501 13:59:21.630075 22735 raft_consensus.cc:2454] T 13a435e74d9e4fd5b3a40fd6483cd460 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.630353 23065 raft_consensus.cc:348] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.630565 23065 raft_consensus.cc:374] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.630659 23065 raft_consensus.cc:729] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.630633 22929 leader_election.cc:302] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.630971 23063 raft_consensus.cc:2781] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.630878 23065 consensus_queue.cc:260] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.631552 23065 ts_tablet_manager.cc:1398] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:21.631844 23065 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.633705 23065 tablet_bootstrap.cc:654] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.634948 23063 raft_consensus.cc:686] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.635558 23063 consensus_queue.cc:237] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.636518 23065 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.636658 23065 ts_tablet_manager.cc:1367] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:21.637346 23065 raft_consensus.cc:348] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.637583 23065 raft_consensus.cc:374] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.637660 23065 raft_consensus.cc:729] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.637928 23065 consensus_queue.cc:260] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.638684 23065 ts_tablet_manager.cc:1398] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.639003 23065 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.641932 23065 tablet_bootstrap.cc:654] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.646535 22593 catalog_manager.cc:5484] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:21.648896 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a435e74d9e4fd5b3a40fd6483cd460" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.648699 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a435e74d9e4fd5b3a40fd6483cd460" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.649284 22866 raft_consensus.cc:2454] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.649657 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
W20240501 13:59:21.651793 22932 leader_election.cc:341] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.651961 22932 leader_election.cc:302] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac; no voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:21.652664 23063 raft_consensus.cc:2726] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:21.653929 23059 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.654088 23059 ts_tablet_manager.cc:1367] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.024s
I20240501 13:59:21.654187 23065 tablet_bootstrap.cc:492] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.654330 23060 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.654343 23065 ts_tablet_manager.cc:1367] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240501 13:59:21.654496 23060 ts_tablet_manager.cc:1367] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.006s
I20240501 13:59:21.654942 23059 raft_consensus.cc:348] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.655176 23059 raft_consensus.cc:374] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.655087 23065 raft_consensus.cc:348] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.655131 23060 raft_consensus.cc:348] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.655278 23065 raft_consensus.cc:374] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.655272 23059 raft_consensus.cc:729] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.655351 23060 raft_consensus.cc:374] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.655361 23065 raft_consensus.cc:729] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.655454 23060 raft_consensus.cc:729] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.655519 23059 consensus_queue.cc:260] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.655565 23065 consensus_queue.cc:260] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.655694 23060 consensus_queue.cc:260] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.656116 23059 ts_tablet_manager.cc:1398] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.656328 23065 ts_tablet_manager.cc:1398] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.656435 23059 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.656620 23065 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.657305 23060 ts_tablet_manager.cc:1398] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:21.657631 23060 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.658166 23059 tablet_bootstrap.cc:654] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.658416 23065 tablet_bootstrap.cc:654] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.658664 23067 raft_consensus.cc:482] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.658814 23067 raft_consensus.cc:504] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.659312 23060 tablet_bootstrap.cc:654] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.659572 23067 leader_election.cc:288] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.660681 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc17872dfe046919f1540dedf2f806c" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.661003 22865 raft_consensus.cc:2454] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.661478 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc17872dfe046919f1540dedf2f806c" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.661783 22996 raft_consensus.cc:2454] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.662621 22671 leader_election.cc:302] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.666666 23065 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.666790 23059 tablet_bootstrap.cc:492] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.666844 23065 ts_tablet_manager.cc:1367] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 13:59:21.666947 23059 ts_tablet_manager.cc:1367] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 13:59:21.667675 23059 raft_consensus.cc:348] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.667673 23065 raft_consensus.cc:348] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.667909 23059 raft_consensus.cc:374] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.667909 23065 raft_consensus.cc:374] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.667980 23059 raft_consensus.cc:729] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.667981 23065 raft_consensus.cc:729] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.668216 23065 consensus_queue.cc:260] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.668216 23059 consensus_queue.cc:260] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.668843 23065 ts_tablet_manager.cc:1398] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.669175 23065 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.669317 23059 ts_tablet_manager.cc:1398] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.669626 23059 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.670150 23067 raft_consensus.cc:2781] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.670301 23067 raft_consensus.cc:482] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.670400 23067 raft_consensus.cc:3037] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.670939 23065 tablet_bootstrap.cc:654] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.671285 23059 tablet_bootstrap.cc:654] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.674360 23060 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.674623 23060 ts_tablet_manager.cc:1367] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.013s
I20240501 13:59:21.674860 23059 tablet_bootstrap.cc:492] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.674984 23059 ts_tablet_manager.cc:1367] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:21.675424 23060 raft_consensus.cc:348] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.675741 23060 raft_consensus.cc:374] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.675824 23060 raft_consensus.cc:729] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.675717 23059 raft_consensus.cc:348] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.675920 23059 raft_consensus.cc:374] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.676013 23059 raft_consensus.cc:729] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.676240 23059 consensus_queue.cc:260] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.676908 23059 ts_tablet_manager.cc:1398] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 13:59:21.677196 23059 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.678238 23060 consensus_queue.cc:260] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.678844 23067 raft_consensus.cc:504] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.679033 23059 tablet_bootstrap.cc:654] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.680096 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc17872dfe046919f1540dedf2f806c" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.680369 22865 raft_consensus.cc:3037] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.680940 23060 ts_tablet_manager.cc:1398] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 13:59:21.681319 23060 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.681394 23065 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.681543 23065 ts_tablet_manager.cc:1367] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240501 13:59:21.682040 23059 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.682191 23059 ts_tablet_manager.cc:1367] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:21.682412 23065 raft_consensus.cc:348] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.682619 23065 raft_consensus.cc:374] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.682713 23065 raft_consensus.cc:729] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.682997 23060 tablet_bootstrap.cc:654] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.682927 23059 raft_consensus.cc:348] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.682994 23065 consensus_queue.cc:260] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.683137 23059 raft_consensus.cc:374] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.683199 23059 raft_consensus.cc:729] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.683437 23059 consensus_queue.cc:260] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.683624 23065 ts_tablet_manager.cc:1398] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.683908 23065 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.686127 23065 tablet_bootstrap.cc:654] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.686417 23059 ts_tablet_manager.cc:1398] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:21.687521 23059 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.689157 23059 tablet_bootstrap.cc:654] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.689886 22865 raft_consensus.cc:2454] T 7dc17872dfe046919f1540dedf2f806c P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.690565 22671 leader_election.cc:302] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.690964 23102 raft_consensus.cc:2781] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.691869 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc17872dfe046919f1540dedf2f806c" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.692131 22996 raft_consensus.cc:3037] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.696393 22996 raft_consensus.cc:2454] T 7dc17872dfe046919f1540dedf2f806c P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.697324 23067 leader_election.cc:288] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.698421 23102 raft_consensus.cc:686] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.699102 23102 consensus_queue.cc:237] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.700537 23063 raft_consensus.cc:482] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.700721 23063 raft_consensus.cc:504] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.700857 23065 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.700881 23060 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.701020 23060 ts_tablet_manager.cc:1367] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20240501 13:59:21.701031 23065 ts_tablet_manager.cc:1367] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240501 13:59:21.701380 23063 leader_election.cc:288] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.701860 23060 raft_consensus.cc:348] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.701828 23065 raft_consensus.cc:348] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.702054 23060 raft_consensus.cc:374] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.702111 23065 raft_consensus.cc:374] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.702188 23065 raft_consensus.cc:729] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.702157 23060 raft_consensus.cc:729] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.702425 23060 consensus_queue.cc:260] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.703006 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b598b69784ab4492957ea8c8e615db3a" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.703155 23060 ts_tablet_manager.cc:1398] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.703233 22735 raft_consensus.cc:2454] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.703444 23060 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.703744 22929 leader_election.cc:302] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.704128 23063 raft_consensus.cc:2781] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.704243 23059 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.704241 23063 raft_consensus.cc:482] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.704465 23059 ts_tablet_manager.cc:1367] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 13:59:21.704452 23063 raft_consensus.cc:3037] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.705330 23060 tablet_bootstrap.cc:654] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.705574 23059 raft_consensus.cc:348] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.706199 23059 raft_consensus.cc:374] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.706323 23059 raft_consensus.cc:729] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.706656 23059 consensus_queue.cc:260] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.707312 23059 ts_tablet_manager.cc:1398] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:21.707579 23059 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.707901 23063 raft_consensus.cc:504] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.709477 23059 tablet_bootstrap.cc:654] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.709597 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b598b69784ab4492957ea8c8e615db3a" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.709921 22735 raft_consensus.cc:3037] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.713608 23060 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.713814 23060 ts_tablet_manager.cc:1367] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 13:59:21.713866 22735 raft_consensus.cc:2454] T b598b69784ab4492957ea8c8e615db3a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.714375 22929 leader_election.cc:302] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.714532 23063 leader_election.cc:288] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.714710 23063 raft_consensus.cc:2781] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.714702 23060 raft_consensus.cc:348] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.714864 23063 raft_consensus.cc:686] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.714942 23060 raft_consensus.cc:374] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.715023 23060 raft_consensus.cc:729] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.715171 23063 consensus_queue.cc:237] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.715246 23060 consensus_queue.cc:260] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.715757 23060 ts_tablet_manager.cc:1398] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:21.716073 23060 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.716364 23065 consensus_queue.cc:260] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.716854 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b598b69784ab4492957ea8c8e615db3a" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.716883 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b598b69784ab4492957ea8c8e615db3a" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.717108 23065 ts_tablet_manager.cc:1398] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 13:59:21.716712 22592 catalog_manager.cc:5484] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.717471 23065 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.717111 22865 raft_consensus.cc:2454] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.719147 23060 tablet_bootstrap.cc:654] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.719147 23065 tablet_bootstrap.cc:654] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.722337 23059 tablet_bootstrap.cc:492] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.722551 23059 ts_tablet_manager.cc:1367] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 13:59:21.723367 23059 raft_consensus.cc:348] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.723626 23059 raft_consensus.cc:374] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.723718 23059 raft_consensus.cc:729] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.723723 23060 tablet_bootstrap.cc:492] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.723870 23060 ts_tablet_manager.cc:1367] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240501 13:59:21.723958 23059 consensus_queue.cc:260] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.724191 23065 tablet_bootstrap.cc:492] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.724454 23065 ts_tablet_manager.cc:1367] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 13:59:21.724524 23059 ts_tablet_manager.cc:1398] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:21.724354 22593 catalog_manager.cc:5484] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:21.724676 23060 raft_consensus.cc:348] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.724835 23059 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.724897 23060 raft_consensus.cc:374] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.724988 23060 raft_consensus.cc:729] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.725226 23060 consensus_queue.cc:260] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.725279 23065 raft_consensus.cc:348] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.725481 23065 raft_consensus.cc:374] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.725566 23065 raft_consensus.cc:729] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.725804 23060 ts_tablet_manager.cc:1398] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.725991 23072 raft_consensus.cc:482] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.726089 23060 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.726153 23072 raft_consensus.cc:504] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.727799 23060 tablet_bootstrap.cc:654] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.728093 23059 tablet_bootstrap.cc:654] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.728286 23072 leader_election.cc:288] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.728601 23065 consensus_queue.cc:260] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.729614 23065 ts_tablet_manager.cc:1398] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:21.729982 23065 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.732343 23065 tablet_bootstrap.cc:654] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.739748 23060 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.739975 23060 ts_tablet_manager.cc:1367] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240501 13:59:21.743427 23065 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.743628 23065 ts_tablet_manager.cc:1367] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20240501 13:59:21.744469 23065 raft_consensus.cc:348] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.744740 23065 raft_consensus.cc:374] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.744830 23065 raft_consensus.cc:729] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.745142 23065 consensus_queue.cc:260] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.745839 23065 ts_tablet_manager.cc:1398] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:21.746179 23065 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4: Bootstrap starting.
I20240501 13:59:21.747588 23102 raft_consensus.cc:482] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.748050 23065 tablet_bootstrap.cc:654] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.748020 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b020e221592443ba62fe794ae418a8a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.747917 23102 raft_consensus.cc:504] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
W20240501 13:59:21.748858 22799 leader_election.cc:341] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.740932 23060 raft_consensus.cc:348] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.749076 23060 raft_consensus.cc:374] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.749202 23060 raft_consensus.cc:729] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.749475 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.749469 23060 consensus_queue.cc:260] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.750205 23060 ts_tablet_manager.cc:1398] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20240501 13:59:21.750283 22669 leader_election.cc:341] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.750627 23060 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.752974 23060 tablet_bootstrap.cc:654] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.753739 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.753999 22865 raft_consensus.cc:2454] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.754575 22671 leader_election.cc:302] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters: 176b379ed1a14fe6ac40373a79c4a6ac
I20240501 13:59:21.754912 23107 raft_consensus.cc:2781] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.755020 23107 raft_consensus.cc:482] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.755136 23107 raft_consensus.cc:3037] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.752997 23102 leader_election.cc:288] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.757799 23102 raft_consensus.cc:482] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.757946 23102 raft_consensus.cc:504] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.758285 23107 raft_consensus.cc:504] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.759127 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0da15dc67d47a48a5ac69927258738" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.759361 22996 raft_consensus.cc:2454] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.759811 22669 leader_election.cc:302] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.760107 23067 raft_consensus.cc:2781] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.760207 23067 raft_consensus.cc:482] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.760322 23067 raft_consensus.cc:3037] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.760257 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0da15dc67d47a48a5ac69927258738" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.760465 22865 raft_consensus.cc:2454] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.761680 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
W20240501 13:59:21.762197 22669 leader_election.cc:341] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.762931 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.763119 22865 raft_consensus.cc:3037] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.763276 23063 raft_consensus.cc:482] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.763304 23067 raft_consensus.cc:504] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.763465 23063 raft_consensus.cc:504] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.763934 23067 leader_election.cc:288] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.764333 23063 leader_election.cc:288] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.764835 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0da15dc67d47a48a5ac69927258738" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.765028 22864 raft_consensus.cc:3037] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.765151 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.765442 22866 raft_consensus.cc:2454] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.766014 22865 raft_consensus.cc:2454] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.766067 22932 leader_election.cc:302] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.766434 22671 leader_election.cc:302] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters: 176b379ed1a14fe6ac40373a79c4a6ac
I20240501 13:59:21.766872 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0da15dc67d47a48a5ac69927258738" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.766985 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.767127 22996 raft_consensus.cc:3037] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:59:21.767494 22929 leader_election.cc:341] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.767891 22864 raft_consensus.cc:2454] T 2f0da15dc67d47a48a5ac69927258738 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.767948 23067 raft_consensus.cc:2781] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.768265 23107 leader_election.cc:288] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.768574 22671 leader_election.cc:302] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.768905 23107 raft_consensus.cc:2781] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.769057 23107 raft_consensus.cc:686] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.769338 23107 consensus_queue.cc:237] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.770244 23063 raft_consensus.cc:2781] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.770408 23063 raft_consensus.cc:482] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.770526 23063 raft_consensus.cc:3037] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.771849 23102 leader_election.cc:288] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.774274 23063 raft_consensus.cc:504] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.775943 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.776131 22996 raft_consensus.cc:2454] T 2f0da15dc67d47a48a5ac69927258738 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
W20240501 13:59:21.776638 22929 leader_election.cc:341] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:21.777658 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.777966 22865 raft_consensus.cc:3037] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.777575 23063 leader_election.cc:288] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.780870 23065 tablet_bootstrap.cc:492] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4: No bootstrap required, opened a new log
I20240501 13:59:21.780997 22865 raft_consensus.cc:2454] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.781407 23067 raft_consensus.cc:686] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.781850 23067 consensus_queue.cc:237] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.781044 23065 ts_tablet_manager.cc:1367] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.008s
I20240501 13:59:21.783253 22932 leader_election.cc:302] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4; no voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5
I20240501 13:59:21.783358 23065 raft_consensus.cc:348] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.783639 23065 raft_consensus.cc:374] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.783735 23065 raft_consensus.cc:729] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.783877 23093 raft_consensus.cc:482] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.784060 23093 raft_consensus.cc:504] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.784143 23065 consensus_queue.cc:260] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.784739 23065 ts_tablet_manager.cc:1398] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.785028 23093 leader_election.cc:288] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.785279 23093 raft_consensus.cc:2781] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.785450 23093 raft_consensus.cc:686] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.785811 23093 consensus_queue.cc:237] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.788421 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.788679 22735 raft_consensus.cc:2454] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.788502 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.788957 22865 raft_consensus.cc:2454] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.789067 23059 tablet_bootstrap.cc:492] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.789253 23059 ts_tablet_manager.cc:1367] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.064s user 0.017s sys 0.008s
I20240501 13:59:21.789284 22929 leader_election.cc:302] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.790069 23059 raft_consensus.cc:348] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.790315 23059 raft_consensus.cc:374] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.790416 23059 raft_consensus.cc:729] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.790810 23059 consensus_queue.cc:260] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.791699 23059 ts_tablet_manager.cc:1398] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:21.791873 23126 raft_consensus.cc:2781] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.791981 23059 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.791983 23126 raft_consensus.cc:482] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.792182 23126 raft_consensus.cc:3037] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.794014 23059 tablet_bootstrap.cc:654] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.795389 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b020e221592443ba62fe794ae418a8a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.795630 23126 raft_consensus.cc:504] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.796691 23126 leader_election.cc:288] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.797605 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.797981 22865 raft_consensus.cc:3037] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.800375 23059 tablet_bootstrap.cc:492] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.800554 23059 ts_tablet_manager.cc:1367] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240501 13:59:21.801383 23059 raft_consensus.cc:348] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.801607 23059 raft_consensus.cc:374] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.801702 23059 raft_consensus.cc:729] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.801931 23059 consensus_queue.cc:260] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.802523 23059 ts_tablet_manager.cc:1398] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:21.795806 22996 raft_consensus.cc:2454] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.802829 23059 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac: Bootstrap starting.
I20240501 13:59:21.803740 23107 raft_consensus.cc:482] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.803886 23107 raft_consensus.cc:504] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.804863 23107 leader_election.cc:288] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.805176 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09d9430f6aa4507a4f16d7796c2965b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.805449 22864 raft_consensus.cc:2454] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.805919 22671 leader_election.cc:302] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.806257 23107 raft_consensus.cc:2781] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.806375 23107 raft_consensus.cc:482] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.806473 23107 raft_consensus.cc:3037] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.806852 23060 tablet_bootstrap.cc:492] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.807107 23060 ts_tablet_manager.cc:1367] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.057s user 0.022s sys 0.034s
I20240501 13:59:21.808177 23060 raft_consensus.cc:348] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.808456 23060 raft_consensus.cc:374] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.808571 23060 raft_consensus.cc:729] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.808614 23107 raft_consensus.cc:504] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.808912 23060 consensus_queue.cc:260] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.809450 23107 leader_election.cc:288] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.809832 23060 ts_tablet_manager.cc:1398] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:21.809881 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09d9430f6aa4507a4f16d7796c2965b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.810101 23060 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.810189 22864 raft_consensus.cc:3037] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.810295 22865 raft_consensus.cc:2454] T a2c272fe58b54fbcb2e4dfda95f46977 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.810760 22932 leader_election.cc:302] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.811097 23063 raft_consensus.cc:2781] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.811232 23063 raft_consensus.cc:686] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.811527 23063 consensus_queue.cc:237] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.811846 23060 tablet_bootstrap.cc:654] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.813585 23059 tablet_bootstrap.cc:654] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.814023 22864 raft_consensus.cc:2454] T b09d9430f6aa4507a4f16d7796c2965b P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.814581 22671 leader_election.cc:302] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.814919 23107 raft_consensus.cc:2781] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.815089 23107 raft_consensus.cc:686] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.815352 23107 consensus_queue.cc:237] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.818396 23060 tablet_bootstrap.cc:492] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.818730 23060 ts_tablet_manager.cc:1367] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240501 13:59:21.819782 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c272fe58b54fbcb2e4dfda95f46977" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.819854 23060 raft_consensus.cc:348] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.820003 22735 raft_consensus.cc:3037] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.820072 23060 raft_consensus.cc:374] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.820158 23060 raft_consensus.cc:729] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.820477 23059 tablet_bootstrap.cc:492] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac: No bootstrap required, opened a new log
I20240501 13:59:21.820441 23060 consensus_queue.cc:260] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.820618 23059 ts_tablet_manager.cc:1367] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 13:59:21.821058 23060 ts_tablet_manager.cc:1398] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:21.820858 22592 catalog_manager.cc:5484] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:21.821419 23060 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Bootstrap starting.
I20240501 13:59:21.821627 23059 raft_consensus.cc:348] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.821838 23059 raft_consensus.cc:374] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.822841 22735 raft_consensus.cc:2454] T a2c272fe58b54fbcb2e4dfda95f46977 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.823112 23060 tablet_bootstrap.cc:654] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.823662 22593 catalog_manager.cc:5484] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.824981 23060 tablet_bootstrap.cc:492] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: No bootstrap required, opened a new log
I20240501 13:59:21.825007 23059 raft_consensus.cc:729] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.825162 23060 ts_tablet_manager.cc:1367] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:21.825306 23059 consensus_queue.cc:260] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:21.825439 22800 leader_election.cc:302] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4; no voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5
I20240501 13:59:21.825799 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09d9430f6aa4507a4f16d7796c2965b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.825991 23059 ts_tablet_manager.cc:1398] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:21.825942 23060 raft_consensus.cc:348] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.826066 22996 raft_consensus.cc:2454] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.826148 23060 raft_consensus.cc:374] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:21.826227 23060 raft_consensus.cc:729] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Initialized, Role: FOLLOWER
I20240501 13:59:21.826260 23072 raft_consensus.cc:2781] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.826376 23072 raft_consensus.cc:482] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.826467 23072 raft_consensus.cc:3037] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.826443 23060 consensus_queue.cc:260] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.827014 23060 ts_tablet_manager.cc:1398] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:21.827217 23133 raft_consensus.cc:482] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.827239 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09d9430f6aa4507a4f16d7796c2965b" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.827401 22996 raft_consensus.cc:3037] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.827431 23133 raft_consensus.cc:504] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.828667 23133 leader_election.cc:288] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.829026 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.829180 23067 raft_consensus.cc:482] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.829267 22735 raft_consensus.cc:2454] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.829344 23067 raft_consensus.cc:504] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.829741 22996 raft_consensus.cc:2454] T b09d9430f6aa4507a4f16d7796c2965b P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.829810 22799 leader_election.cc:302] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.829749 23072 raft_consensus.cc:504] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.830121 23067 leader_election.cc:288] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.830569 23133 raft_consensus.cc:2781] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.830664 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c2dcdd384e484d8f95f47109f19e69" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.830744 23133 raft_consensus.cc:482] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.830847 23133 raft_consensus.cc:3037] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.830864 22996 raft_consensus.cc:2454] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.831054 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c2dcdd384e484d8f95f47109f19e69" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.831295 22864 raft_consensus.cc:2454] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.831653 22669 leader_election.cc:302] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.831929 23067 raft_consensus.cc:2781] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.832033 23067 raft_consensus.cc:482] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.832118 23067 raft_consensus.cc:3037] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.833720 23133 raft_consensus.cc:504] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.834283 23093 raft_consensus.cc:482] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.834200 23067 raft_consensus.cc:504] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.834427 23093 raft_consensus.cc:504] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.834983 23067 leader_election.cc:288] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.835062 23093 leader_election.cc:288] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.835487 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c2dcdd384e484d8f95f47109f19e69" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.835675 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.835752 22864 raft_consensus.cc:3037] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.835649 22592 catalog_manager.cc:5484] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:21.836246 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.836432 22735 raft_consensus.cc:2454] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.836819 22929 leader_election.cc:302] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.837237 23093 raft_consensus.cc:2781] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.837319 23093 raft_consensus.cc:482] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.837388 23093 raft_consensus.cc:3037] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.838325 22864 raft_consensus.cc:2454] T 73c2dcdd384e484d8f95f47109f19e69 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.838799 22671 leader_election.cc:302] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.839929 23093 raft_consensus.cc:504] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.840102 23072 leader_election.cc:288] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.840468 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b020e221592443ba62fe794ae418a8a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.840664 22735 raft_consensus.cc:3037] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.840853 23093 leader_election.cc:288] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:21.840910 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.841086 22734 raft_consensus.cc:3037] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.841143 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.841949 22865 raft_consensus.cc:2379] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 176b379ed1a14fe6ac40373a79c4a6ac in current term 1: Already voted for candidate 49e10d94bfc743e69509789884c0e3f4 in this term.
I20240501 13:59:21.841742 22592 catalog_manager.cc:5484] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.842365 22592 catalog_manager.cc:5484] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.843746 23133 leader_election.cc:288] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.843873 22734 raft_consensus.cc:2454] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.844051 23067 raft_consensus.cc:2781] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.844215 23067 raft_consensus.cc:686] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.844511 23067 consensus_queue.cc:237] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.844771 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.845033 22734 raft_consensus.cc:2379] T 3c5445f1dc614c41ac4bd83d1f249d82 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e10d94bfc743e69509789884c0e3f4 in current term 1: Already voted for candidate 176b379ed1a14fe6ac40373a79c4a6ac in this term.
I20240501 13:59:21.844974 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c2dcdd384e484d8f95f47109f19e69" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.845149 22735 raft_consensus.cc:2454] T 5b020e221592443ba62fe794ae418a8a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.845275 22996 raft_consensus.cc:3037] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.846719 22799 leader_election.cc:302] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.847024 23133 raft_consensus.cc:2781] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.847218 23133 raft_consensus.cc:686] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Running, Role: LEADER
I20240501 13:59:21.847414 22995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.847793 23133 consensus_queue.cc:237] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.847960 22995 raft_consensus.cc:2379] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49e10d94bfc743e69509789884c0e3f4 in current term 1: Already voted for candidate 176b379ed1a14fe6ac40373a79c4a6ac in this term.
I20240501 13:59:21.848017 22929 leader_election.cc:302] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters: 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:21.847550 22994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b020e221592443ba62fe794ae418a8a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.848351 23093 raft_consensus.cc:2781] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.848220 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5445f1dc614c41ac4bd83d1f249d82" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.848444 22994 raft_consensus.cc:3037] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.848507 23093 raft_consensus.cc:686] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.848771 23093 consensus_queue.cc:237] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.849839 22592 catalog_manager.cc:5484] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.851580 22994 raft_consensus.cc:2454] T 5b020e221592443ba62fe794ae418a8a P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.853564 22996 raft_consensus.cc:2454] T 73c2dcdd384e484d8f95f47109f19e69 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.855150 22800 leader_election.cc:302] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e10d94bfc743e69509789884c0e3f4; no voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5
I20240501 13:59:21.855464 23072 raft_consensus.cc:2726] T 3c5445f1dc614c41ac4bd83d1f249d82 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:59:21.856415 22592 catalog_manager.cc:5484] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:21.857049 22593 catalog_manager.cc:5484] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "49e10d94bfc743e69509789884c0e3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.872300 23133 raft_consensus.cc:482] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.872547 23133 raft_consensus.cc:504] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.873196 23133 leader_election.cc:288] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.873818 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87be1a7f20ab421da40eaa3aed61938c" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.873840 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87be1a7f20ab421da40eaa3aed61938c" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.874070 22734 raft_consensus.cc:2454] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.874091 22993 raft_consensus.cc:2454] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.874491 22799 leader_election.cc:302] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.874784 23133 raft_consensus.cc:2781] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.874874 23133 raft_consensus.cc:482] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.875005 23133 raft_consensus.cc:3037] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.876850 23133 raft_consensus.cc:504] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.877460 23133 leader_election.cc:288] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.877905 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87be1a7f20ab421da40eaa3aed61938c" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.877944 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87be1a7f20ab421da40eaa3aed61938c" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.878111 22993 raft_consensus.cc:3037] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.878115 22734 raft_consensus.cc:3037] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.879969 22734 raft_consensus.cc:2454] T 87be1a7f20ab421da40eaa3aed61938c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.880072 22993 raft_consensus.cc:2454] T 87be1a7f20ab421da40eaa3aed61938c P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.880427 22799 leader_election.cc:302] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.880709 23133 raft_consensus.cc:2781] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.880884 23133 raft_consensus.cc:686] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Running, Role: LEADER
I20240501 13:59:21.881155 23133 consensus_queue.cc:237] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.883849 22593 catalog_manager.cc:5484] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "49e10d94bfc743e69509789884c0e3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.895156 23133 raft_consensus.cc:482] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.895372 23133 raft_consensus.cc:504] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.896056 23133 leader_election.cc:288] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.896623 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8ea9c9db814093a45e2f4d0e4a5e3b" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.896678 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8ea9c9db814093a45e2f4d0e4a5e3b" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.896865 22734 raft_consensus.cc:2454] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.896914 22993 raft_consensus.cc:2454] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:21.897292 22799 leader_election.cc:302] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.897586 23133 raft_consensus.cc:2781] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.897702 23133 raft_consensus.cc:482] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.897789 23133 raft_consensus.cc:3037] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.899606 23133 raft_consensus.cc:504] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.900172 23133 leader_election.cc:288] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.900641 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8ea9c9db814093a45e2f4d0e4a5e3b" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.900657 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8ea9c9db814093a45e2f4d0e4a5e3b" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.900848 22993 raft_consensus.cc:3037] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.900847 22734 raft_consensus.cc:3037] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.902786 22734 raft_consensus.cc:2454] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.902791 22993 raft_consensus.cc:2454] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:21.903244 22799 leader_election.cc:302] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.903585 23133 raft_consensus.cc:2781] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.903748 23133 raft_consensus.cc:686] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Running, Role: LEADER
I20240501 13:59:21.904059 23133 consensus_queue.cc:237] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.904487 23067 raft_consensus.cc:482] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.904659 23067 raft_consensus.cc:504] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.905388 23067 leader_election.cc:288] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.905877 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:21.906088 22993 raft_consensus.cc:2454] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.906497 22669 leader_election.cc:302] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.906491 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.906688 22865 raft_consensus.cc:2454] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 0.
I20240501 13:59:21.907156 22593 catalog_manager.cc:5484] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "49e10d94bfc743e69509789884c0e3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.908139 23067 raft_consensus.cc:2781] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.908226 23067 raft_consensus.cc:482] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.908339 23067 raft_consensus.cc:3037] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.910316 23067 raft_consensus.cc:504] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.910923 23067 leader_election.cc:288] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:21.911309 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.911409 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3610a4cab54c2caf976751927b1eb7" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:21.911509 22865 raft_consensus.cc:3037] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.911631 22993 raft_consensus.cc:3037] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.913458 22865 raft_consensus.cc:2454] T 1b3610a4cab54c2caf976751927b1eb7 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.913473 22993 raft_consensus.cc:2454] T 1b3610a4cab54c2caf976751927b1eb7 P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in term 1.
I20240501 13:59:21.913869 22669 leader_election.cc:302] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.914168 23067 raft_consensus.cc:2781] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.914372 23067 raft_consensus.cc:686] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3750f428b8f84fbdb3fe4d01fb1fe9d5, State: Running, Role: LEADER
I20240501 13:59:21.914629 23067 consensus_queue.cc:237] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.917065 22593 catalog_manager.cc:5484] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:21.980012 23093 raft_consensus.cc:482] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.980255 23093 raft_consensus.cc:504] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.980901 23093 leader_election.cc:288] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975)
I20240501 13:59:21.981524 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:21.981523 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:21.981729 22865 raft_consensus.cc:2454] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.981747 22734 raft_consensus.cc:2454] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 0.
I20240501 13:59:21.982121 22932 leader_election.cc:302] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:21.982414 23093 raft_consensus.cc:2781] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.982497 23093 raft_consensus.cc:482] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.982573 23093 raft_consensus.cc:3037] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.984354 23093 raft_consensus.cc:504] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.984939 23093 leader_election.cc:288] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Requested vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975)
I20240501 13:59:21.985482 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:21.985486 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a255d18518e64e789f16f4dd14afcbdc" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4"
I20240501 13:59:21.985672 22865 raft_consensus.cc:3037] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.985666 22734 raft_consensus.cc:3037] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.987507 22734 raft_consensus.cc:2454] T a255d18518e64e789f16f4dd14afcbdc P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.987517 22865 raft_consensus.cc:2454] T a255d18518e64e789f16f4dd14afcbdc P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176b379ed1a14fe6ac40373a79c4a6ac in term 1.
I20240501 13:59:21.987969 22929 leader_election.cc:302] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters:
I20240501 13:59:21.988271 23093 raft_consensus.cc:2781] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.988451 23093 raft_consensus.cc:686] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 LEADER]: Becoming Leader. State: Replica: 176b379ed1a14fe6ac40373a79c4a6ac, State: Running, Role: LEADER
I20240501 13:59:21.988706 23093 consensus_queue.cc:237] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:21.991433 22593 catalog_manager.cc:5484] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195). New cstate: current_term: 1 leader_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:22.014431 22865 raft_consensus.cc:3037] T 13a435e74d9e4fd5b3a40fd6483cd460 P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.014482 23093 raft_consensus.cc:482] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.014772 23093 raft_consensus.cc:504] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } }
I20240501 13:59:22.015498 23093 leader_election.cc:288] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259)
I20240501 13:59:22.016014 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:22.016083 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4a9b798b5e4a308a59bd8d45811c5b" candidate_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:22.016386 22864 raft_consensus.cc:2379] T 2b4a9b798b5e4a308a59bd8d45811c5b P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 176b379ed1a14fe6ac40373a79c4a6ac in current term 1: Already voted for candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in this term.
I20240501 13:59:22.016769 22932 leader_election.cc:302] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac; no voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:22.017062 23093 raft_consensus.cc:3037] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.018177 23125 consensus_queue.cc:1035] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.019551 23093 raft_consensus.cc:2726] T 2b4a9b798b5e4a308a59bd8d45811c5b P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:22.029191 23126 consensus_queue.cc:1035] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.076913 23133 raft_consensus.cc:482] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.077260 23133 raft_consensus.cc:504] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:22.077936 23133 leader_election.cc:288] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:22.078541 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dac4f984e54979b432ebde9a9b5d0a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" is_pre_election: true
I20240501 13:59:22.078583 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dac4f984e54979b432ebde9a9b5d0a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:22.078745 22734 raft_consensus.cc:2454] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:22.078816 22993 raft_consensus.cc:2454] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 0.
I20240501 13:59:22.079192 22799 leader_election.cc:302] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:22.079520 23133 raft_consensus.cc:2781] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:22.079627 23133 raft_consensus.cc:482] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:22.079718 23133 raft_consensus.cc:3037] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.081746 23133 raft_consensus.cc:504] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:22.082353 23133 leader_election.cc:288] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:22.082849 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dac4f984e54979b432ebde9a9b5d0a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5"
I20240501 13:59:22.082901 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dac4f984e54979b432ebde9a9b5d0a" candidate_uuid: "49e10d94bfc743e69509789884c0e3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac"
I20240501 13:59:22.083046 22734 raft_consensus.cc:3037] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.083087 22993 raft_consensus.cc:3037] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.084928 22734 raft_consensus.cc:2454] T 67dac4f984e54979b432ebde9a9b5d0a P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:22.084931 22993 raft_consensus.cc:2454] T 67dac4f984e54979b432ebde9a9b5d0a P 176b379ed1a14fe6ac40373a79c4a6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e10d94bfc743e69509789884c0e3f4 in term 1.
I20240501 13:59:22.085400 22800 leader_election.cc:302] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 176b379ed1a14fe6ac40373a79c4a6ac, 3750f428b8f84fbdb3fe4d01fb1fe9d5, 49e10d94bfc743e69509789884c0e3f4; no voters:
I20240501 13:59:22.085673 23133 raft_consensus.cc:2781] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:22.085829 23133 raft_consensus.cc:686] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 49e10d94bfc743e69509789884c0e3f4, State: Running, Role: LEADER
I20240501 13:59:22.086091 23133 consensus_queue.cc:237] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:22.088817 22593 catalog_manager.cc:5484] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "49e10d94bfc743e69509789884c0e3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:22.099741 22865 raft_consensus.cc:3037] T b598b69784ab4492957ea8c8e615db3a P 49e10d94bfc743e69509789884c0e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.102705 23125 consensus_queue.cc:1035] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.107877 23093 consensus_queue.cc:1035] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.168154 23067 consensus_queue.cc:1035] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.173199 23067 consensus_queue.cc:1035] T 2b4a9b798b5e4a308a59bd8d45811c5b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.185236 23067 consensus_queue.cc:1035] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.189607 23121 consensus_queue.cc:1035] T 2f0da15dc67d47a48a5ac69927258738 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.197794 23121 consensus_queue.cc:1035] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.203121 23102 consensus_queue.cc:1035] T b09d9430f6aa4507a4f16d7796c2965b P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.204978 23102 consensus_queue.cc:1035] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.209982 23121 consensus_queue.cc:1035] T 7dc17872dfe046919f1540dedf2f806c P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.222510 23093 consensus_queue.cc:1035] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.227190 23063 consensus_queue.cc:1035] T a2c272fe58b54fbcb2e4dfda95f46977 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.246503 23063 consensus_queue.cc:1035] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.251345 23125 consensus_queue.cc:1035] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.263985 23121 consensus_queue.cc:1035] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.269928 23121 consensus_queue.cc:1035] T 73c2dcdd384e484d8f95f47109f19e69 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.303841 23121 raft_consensus.cc:482] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.304127 23121 raft_consensus.cc:504] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 } } peers { permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 } } peers { permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 } }
I20240501 13:59:22.304960 23121 leader_election.cc:288] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259), 176b379ed1a14fe6ac40373a79c4a6ac (127.22.5.195:40325)
I20240501 13:59:22.305425 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e10d94bfc743e69509789884c0e3f4" is_pre_election: true
I20240501 13:59:22.305501 22734 raft_consensus.cc:3037] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.305709 22865 raft_consensus.cc:2379] T b8edade9a59f404e903fdc51cb13a708 P 49e10d94bfc743e69509789884c0e3f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3750f428b8f84fbdb3fe4d01fb1fe9d5 in current term 1: Already voted for candidate 176b379ed1a14fe6ac40373a79c4a6ac in this term.
I20240501 13:59:22.305682 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edade9a59f404e903fdc51cb13a708" candidate_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" is_pre_election: true
I20240501 13:59:22.306625 22669 leader_election.cc:302] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3750f428b8f84fbdb3fe4d01fb1fe9d5; no voters: 176b379ed1a14fe6ac40373a79c4a6ac, 49e10d94bfc743e69509789884c0e3f4
I20240501 13:59:22.309274 23121 raft_consensus.cc:2726] T b8edade9a59f404e903fdc51cb13a708 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:22.309736 23125 consensus_queue.cc:1035] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.314083 23125 consensus_queue.cc:1035] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.321249 23133 consensus_queue.cc:1035] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.327298 23072 consensus_queue.cc:1035] T 5b020e221592443ba62fe794ae418a8a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.368577 23072 consensus_queue.cc:1035] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.396437 23072 consensus_queue.cc:1035] T 87be1a7f20ab421da40eaa3aed61938c P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.420899 23121 consensus_queue.cc:1035] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.424187 23072 consensus_queue.cc:1035] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.425109 23072 consensus_queue.cc:1035] T 0f8ea9c9db814093a45e2f4d0e4a5e3b P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.430768 23063 consensus_queue.cc:1035] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.434998 23093 consensus_queue.cc:1035] T a255d18518e64e789f16f4dd14afcbdc P 176b379ed1a14fe6ac40373a79c4a6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.446472 23107 consensus_queue.cc:1035] T 1b3610a4cab54c2caf976751927b1eb7 P 3750f428b8f84fbdb3fe4d01fb1fe9d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e10d94bfc743e69509789884c0e3f4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.508934 23176 consensus_queue.cc:1035] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3750f428b8f84fbdb3fe4d01fb1fe9d5" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.519143 23072 consensus_queue.cc:1035] T 67dac4f984e54979b432ebde9a9b5d0a P 49e10d94bfc743e69509789884c0e3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176b379ed1a14fe6ac40373a79c4a6ac" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:22.621546 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:22.621883 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:22.629413 23160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 392.735 ms
time per row: 0.196368 ms
I20240501 13:59:23.711206 22976 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:23.712030 22704 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:23.712432 22834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:23.886448 22976 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:23.889014 22704 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:23.889252 22834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:24.093794 22976 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:24.094522 22704 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:24.094935 22834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:24.765673 22976 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:24.770596 22834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:24.773226 22704 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:24.944422 22976 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:24.944571 22834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:24.944713 22704 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:25.102954 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 22653
I20240501 13:59:25.109565 22775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:25.238384 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 22653
I20240501 13:59:25.261313 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 22784
I20240501 13:59:25.267393 22906 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 13:59:25.280112 22929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111)
W20240501 13:59:25.281673 22929 consensus_peers.cc:485] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Couldn't send request to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:25.353366 22929 consensus_peers.cc:485] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Couldn't send request to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:25.382834 22929 consensus_peers.cc:485] T 13a435e74d9e4fd5b3a40fd6483cd460 P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Couldn't send request to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:25.393827 22799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111)
I20240501 13:59:25.393931 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 22784
W20240501 13:59:25.407059 22929 consensus_peers.cc:485] T b598b69784ab4492957ea8c8e615db3a P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 3750f428b8f84fbdb3fe4d01fb1fe9d5 (127.22.5.193:38975): Couldn't send request to peer 3750f428b8f84fbdb3fe4d01fb1fe9d5. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:25.417593 22932 connection.cc:535] client connection to 127.22.5.194:39259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:25.417927 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 22914
W20240501 13:59:25.418953 22932 consensus_peers.cc:485] T 3c5445f1dc614c41ac4bd83d1f249d82 P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259): Couldn't send request to peer 49e10d94bfc743e69509789884c0e3f4. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:39259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:25.419082 22932 consensus_peers.cc:485] T b8edade9a59f404e903fdc51cb13a708 P 176b379ed1a14fe6ac40373a79c4a6ac -> Peer 49e10d94bfc743e69509789884c0e3f4 (127.22.5.194:39259): Couldn't send request to peer 49e10d94bfc743e69509789884c0e3f4. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:39259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:25.423720 23036 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:25.548570 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 22914
I20240501 13:59:25.569849 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 22563
I20240501 13:59:25.570992 22624 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:25.690021 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 22563
2024-05-01T13:59:25Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (10261 ms)
[----------] 1 test from TableRangeRebalancingTest (10261 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240501 13:59:25.744053 22551 test_util.cc:268] Using random seed: -453655653
I20240501 13:59:25.746035 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:25.746101 22551 ts_itest-base.cc:113] --------------
I20240501 13:59:25.746166 22551 ts_itest-base.cc:114] 5 tablet servers
I20240501 13:59:25.746215 22551 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:25.746284 22551 ts_itest-base.cc:116] --------------
2024-05-01T13:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:25Z Disabled control of system clock
I20240501 13:59:25.770104 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:38759
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:38759
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 13:59:26.098691 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:26.099040 23426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:26.099184 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:26.106681 23426 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:26.106839 23426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:26.106920 23426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:26.107005 23426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:26.107074 23426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:26.116753 23426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:38759
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:26.118243 23426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:26.120136 23426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:26.131839 23426 server_base.cc:1034] running on GCE node
W20240501 13:59:26.132226 23432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:26.132925 23434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:26.133006 23431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:26.133908 23426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:26.135335 23426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:26.136554 23426 hybrid_clock.cc:648] HybridClock initialized: now 1714571966136500 us; error 46 us; skew 500 ppm
I20240501 13:59:26.136901 23426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:26.139217 23426 webserver.cc:460] Webserver started at http://127.22.5.254:37657/ using document root <none> and password file <none>
I20240501 13:59:26.139948 23426 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:26.140062 23426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:26.140376 23426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:26.143062 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5fde39e0aedc4af789b89948c8631fcc"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.143819 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5fde39e0aedc4af789b89948c8631fcc"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.149061 23426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:26.152568 23440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:26.153982 23426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:26.154155 23426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5fde39e0aedc4af789b89948c8631fcc"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.154352 23426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:26.172204 23426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:26.173092 23426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:26.173404 23426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:26.191159 23426 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:38759
I20240501 13:59:26.191198 23491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:38759 every 8 connection(s)
I20240501 13:59:26.192553 23426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:26.196193 23492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.197866 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 23426
I20240501 13:59:26.198050 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:26.208118 23492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc: Bootstrap starting.
I20240501 13:59:26.211980 23492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.213052 23492 log.cc:824] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:26.215615 23492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc: No bootstrap required, opened a new log
I20240501 13:59:26.221802 23492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } }
I20240501 13:59:26.222148 23492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.222254 23492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fde39e0aedc4af789b89948c8631fcc, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.222879 23492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } }
I20240501 13:59:26.223091 23492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:26.223223 23492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:26.223407 23492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.225555 23492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } }
I20240501 13:59:26.226040 23492 leader_election.cc:302] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fde39e0aedc4af789b89948c8631fcc; no voters:
I20240501 13:59:26.226581 23492 leader_election.cc:288] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:26.226733 23497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:26.227330 23497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [term 1 LEADER]: Becoming Leader. State: Replica: 5fde39e0aedc4af789b89948c8631fcc, State: Running, Role: LEADER
I20240501 13:59:26.227931 23497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } }
I20240501 13:59:26.228545 23492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:26.230722 23498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fde39e0aedc4af789b89948c8631fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } } }
I20240501 13:59:26.231201 23498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:26.231245 23499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fde39e0aedc4af789b89948c8631fcc. Latest consensus state: current_term: 1 leader_uuid: "5fde39e0aedc4af789b89948c8631fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fde39e0aedc4af789b89948c8631fcc" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 38759 } } }
I20240501 13:59:26.231484 23499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:26.232815 23504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:26.237864 23504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:26.246320 23504 catalog_manager.cc:1347] Generated new cluster ID: 4ed9282cc5274daf9fac0e424aeba695
I20240501 13:59:26.246443 23504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:26.257809 23504 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:26.259151 23504 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:26.266387 23504 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5fde39e0aedc4af789b89948c8631fcc: Generated new TSK 0
I20240501 13:59:26.267210 23504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:26.275089 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:26.655501 23516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:26.655813 23516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:26.655951 23516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:26.663630 23516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:26.663746 23516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:26.663892 23516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 13:59:26.673053 23516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:26.674679 23516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:26.676448 23516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:26.688762 23522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:26.689323 23524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:26.690052 23521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:26.690737 23516 server_base.cc:1034] running on GCE node
I20240501 13:59:26.691264 23516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:26.692162 23516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:26.693358 23516 hybrid_clock.cc:648] HybridClock initialized: now 1714571966693296 us; error 56 us; skew 500 ppm
I20240501 13:59:26.693658 23516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:26.695955 23516 webserver.cc:460] Webserver started at http://127.22.5.193:36431/ using document root <none> and password file <none>
I20240501 13:59:26.696720 23516 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:26.696818 23516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:26.697185 23516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:26.699745 23516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e6629f605fa14313ac9d5fe78ce15563"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.700501 23516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e6629f605fa14313ac9d5fe78ce15563"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.705569 23516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 13:59:26.708834 23530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:26.710270 23516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:26.710422 23516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e6629f605fa14313ac9d5fe78ce15563"
format_stamp: "Formatted at 2024-05-01 13:59:26 on dist-test-slave-5jzj"
I20240501 13:59:26.710614 23516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:26.735509 23516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:26.736438 23516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:26.736758 23516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:26.737772 23516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:26.739287 23516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:26.739390 23516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:26.739509 23516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:26.739581 23516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:26.770622 23516 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:36127
I20240501 13:59:26.770691 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:36127 every 8 connection(s)
I20240501 13:59:26.772025 23516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:26.777709 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 23516
I20240501 13:59:26.777904 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:26.781445 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:26.785259 23643 heartbeater.cc:347] Connected to a master server at 127.22.5.254:38759
I20240501 13:59:26.785589 23643 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:26.786434 23643 heartbeater.cc:511] Master 127.22.5.254:38759 requested a full tablet report, sending...
I20240501 13:59:26.788794 23457 ts_manager.cc:194] Registered new tserver with Master: e6629f605fa14313ac9d5fe78ce15563 (127.22.5.193:36127)
I20240501 13:59:26.791016 23457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:34673
W20240501 13:59:27.116446 23647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:27.116776 23647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:27.116881 23647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:27.124399 23647 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:27.124534 23647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:27.124650 23647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 13:59:27.133865 23647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:27.135247 23647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:27.137049 23647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:27.149458 23655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:27.150048 23653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:27.150712 23652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:27.151341 23647 server_base.cc:1034] running on GCE node
I20240501 13:59:27.151957 23647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:27.152909 23647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:27.154145 23647 hybrid_clock.cc:648] HybridClock initialized: now 1714571967154084 us; error 49 us; skew 500 ppm
I20240501 13:59:27.154511 23647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:27.156687 23647 webserver.cc:460] Webserver started at http://127.22.5.194:37703/ using document root <none> and password file <none>
I20240501 13:59:27.157495 23647 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:27.157609 23647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:27.157933 23647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:27.160696 23647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7bef279d2a404ab0a89d93cc7d8eaab6"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.161504 23647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7bef279d2a404ab0a89d93cc7d8eaab6"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.166473 23647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 13:59:27.169730 23661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.171027 23647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:27.171169 23647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7bef279d2a404ab0a89d93cc7d8eaab6"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.171348 23647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:27.184463 23647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:27.185271 23647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:27.185534 23647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:27.186414 23647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:27.187785 23647 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:27.187866 23647 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.187974 23647 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:27.188047 23647 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.218477 23647 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:41701
I20240501 13:59:27.218494 23773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:41701 every 8 connection(s)
I20240501 13:59:27.219786 23647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:27.221608 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 23647
I20240501 13:59:27.221768 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:27.225104 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:27.233004 23774 heartbeater.cc:347] Connected to a master server at 127.22.5.254:38759
I20240501 13:59:27.233356 23774 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:27.234361 23774 heartbeater.cc:511] Master 127.22.5.254:38759 requested a full tablet report, sending...
I20240501 13:59:27.235919 23457 ts_manager.cc:194] Registered new tserver with Master: 7bef279d2a404ab0a89d93cc7d8eaab6 (127.22.5.194:41701)
I20240501 13:59:27.236662 23457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:34877
W20240501 13:59:27.561077 23777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:27.561421 23777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:27.561568 23777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:27.568953 23777 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:27.569108 23777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:27.569288 23777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 13:59:27.578523 23777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:27.580327 23777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:27.582232 23777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:27.595037 23782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:27.595687 23783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:27.596146 23785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:27.596787 23777 server_base.cc:1034] running on GCE node
I20240501 13:59:27.597437 23777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:27.598421 23777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:27.599638 23777 hybrid_clock.cc:648] HybridClock initialized: now 1714571967599588 us; error 32 us; skew 500 ppm
I20240501 13:59:27.599956 23777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:27.602234 23777 webserver.cc:460] Webserver started at http://127.22.5.195:41345/ using document root <none> and password file <none>
I20240501 13:59:27.602965 23777 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:27.603073 23777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:27.603401 23777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:27.606105 23777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1dd3358187ed4fb687306945e9bbeed2"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.606894 23777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1dd3358187ed4fb687306945e9bbeed2"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.612241 23777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 13:59:27.616030 23791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.617607 23777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:27.617777 23777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1dd3358187ed4fb687306945e9bbeed2"
format_stamp: "Formatted at 2024-05-01 13:59:27 on dist-test-slave-5jzj"
I20240501 13:59:27.617982 23777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:27.637488 23777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:27.638366 23777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:27.638657 23777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:27.639622 23777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:27.641148 23777 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:27.641247 23777 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.641366 23777 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:27.641438 23777 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:27.672693 23777 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:39885
I20240501 13:59:27.672744 23903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:39885 every 8 connection(s)
I20240501 13:59:27.674075 23777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:27.674796 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 23777
I20240501 13:59:27.674978 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:27.678347 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:27.693570 23904 heartbeater.cc:347] Connected to a master server at 127.22.5.254:38759
I20240501 13:59:27.693993 23904 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:27.695276 23904 heartbeater.cc:511] Master 127.22.5.254:38759 requested a full tablet report, sending...
I20240501 13:59:27.703765 23457 ts_manager.cc:194] Registered new tserver with Master: 1dd3358187ed4fb687306945e9bbeed2 (127.22.5.195:39885)
I20240501 13:59:27.704963 23457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:40649
I20240501 13:59:27.794227 23643 heartbeater.cc:503] Master 127.22.5.254:38759 was elected leader, sending a full tablet report...
W20240501 13:59:28.019660 23907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.019979 23907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:28.020084 23907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.027791 23907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:28.027922 23907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:28.028057 23907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 13:59:28.037700 23907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:28.039098 23907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:28.040829 23907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:28.053263 23912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.053743 23913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.054232 23915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:28.055016 23907 server_base.cc:1034] running on GCE node
I20240501 13:59:28.055681 23907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:28.056733 23907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:28.057955 23907 hybrid_clock.cc:648] HybridClock initialized: now 1714571968057903 us; error 49 us; skew 500 ppm
I20240501 13:59:28.058295 23907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:28.060765 23907 webserver.cc:460] Webserver started at http://127.22.5.196:44653/ using document root <none> and password file <none>
I20240501 13:59:28.061604 23907 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:28.061722 23907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:28.062048 23907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:28.066955 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3747dabbc716432fb2b1939c585f4314"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.067817 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3747dabbc716432fb2b1939c585f4314"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.073413 23907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20240501 13:59:28.077060 23921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.078465 23907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:28.078624 23907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3747dabbc716432fb2b1939c585f4314"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.078824 23907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:28.100804 23907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:28.101879 23907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:28.102182 23907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:28.103241 23907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:28.104866 23907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:28.104951 23907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.105068 23907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:28.105167 23907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.137351 23907 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:43617
I20240501 13:59:28.137409 24033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:43617 every 8 connection(s)
I20240501 13:59:28.138713 23907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:28.148061 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 23907
I20240501 13:59:28.148241 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:28.151449 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.197:0
--local_ip_for_outbound_sockets=127.22.5.197
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--builtin_ntp_servers=127.22.5.212:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:28.151585 24034 heartbeater.cc:347] Connected to a master server at 127.22.5.254:38759
I20240501 13:59:28.151870 24034 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:28.152731 24034 heartbeater.cc:511] Master 127.22.5.254:38759 requested a full tablet report, sending...
I20240501 13:59:28.154246 23457 ts_manager.cc:194] Registered new tserver with Master: 3747dabbc716432fb2b1939c585f4314 (127.22.5.196:43617)
I20240501 13:59:28.155398 23457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:57993
I20240501 13:59:28.239859 23774 heartbeater.cc:503] Master 127.22.5.254:38759 was elected leader, sending a full tablet report...
W20240501 13:59:28.491072 24037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.491364 24037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:28.491467 24037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.498883 24037 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:28.499003 24037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:28.499120 24037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.197
I20240501 13:59:28.508142 24037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:38759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.197
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:28.509517 24037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:28.511224 24037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:28.523557 24042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.524232 24043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.524853 24045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:28.525522 24037 server_base.cc:1034] running on GCE node
I20240501 13:59:28.526197 24037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:28.527210 24037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:28.528426 24037 hybrid_clock.cc:648] HybridClock initialized: now 1714571968528357 us; error 44 us; skew 500 ppm
I20240501 13:59:28.528786 24037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:28.531241 24037 webserver.cc:460] Webserver started at http://127.22.5.197:36595/ using document root <none> and password file <none>
I20240501 13:59:28.532037 24037 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:28.532145 24037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:28.532471 24037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:28.535252 24037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "12a507c6ded2492099ac01fba43a24be"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.536070 24037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "12a507c6ded2492099ac01fba43a24be"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.541555 24037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:28.545017 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.546437 24037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:28.546588 24037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "12a507c6ded2492099ac01fba43a24be"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-5jzj"
I20240501 13:59:28.546808 24037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:28.565985 24037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:28.566902 24037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:28.567178 24037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:28.568132 24037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:28.569795 24037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:28.569880 24037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.569988 24037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:28.570056 24037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.600843 24037 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.197:41625
I20240501 13:59:28.600945 24163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.197:41625 every 8 connection(s)
I20240501 13:59:28.602250 24037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:28.609750 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24037
I20240501 13:59:28.609952 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:28.615023 24164 heartbeater.cc:347] Connected to a master server at 127.22.5.254:38759
I20240501 13:59:28.615365 24164 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:28.616190 24164 heartbeater.cc:511] Master 127.22.5.254:38759 requested a full tablet report, sending...
I20240501 13:59:28.617621 23457 ts_manager.cc:194] Registered new tserver with Master: 12a507c6ded2492099ac01fba43a24be (127.22.5.197:41625)
I20240501 13:59:28.618340 23457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.197:59899
I20240501 13:59:28.625722 22551 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:28.642035 23457 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40052:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:28.687805 23709 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b (DEFAULT_TABLE table=TestTable [id=21ab955dfca14469913874891bc71189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.689092 23709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.690356 23578 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b (DEFAULT_TABLE table=TestTable [id=21ab955dfca14469913874891bc71189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.690356 24099 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b (DEFAULT_TABLE table=TestTable [id=21ab955dfca14469913874891bc71189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:28.691599 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.691633 24099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3436c0b6834bfabbe2d7f7a0a1b99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.699167 24182 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Bootstrap starting.
I20240501 13:59:28.702368 24182 tablet_bootstrap.cc:654] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.703286 24182 log.cc:824] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:28.704432 24184 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Bootstrap starting.
I20240501 13:59:28.705447 24183 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Bootstrap starting.
I20240501 13:59:28.707839 23904 heartbeater.cc:503] Master 127.22.5.254:38759 was elected leader, sending a full tablet report...
I20240501 13:59:28.708931 24184 tablet_bootstrap.cc:654] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.708931 24183 tablet_bootstrap.cc:654] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.709864 24182 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: No bootstrap required, opened a new log
I20240501 13:59:28.710172 24182 ts_tablet_manager.cc:1367] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240501 13:59:28.710194 24184 log.cc:824] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:28.710600 24183 log.cc:824] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:28.713549 24183 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: No bootstrap required, opened a new log
I20240501 13:59:28.713768 24183 ts_tablet_manager.cc:1367] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:59:28.714041 24184 tablet_bootstrap.cc:492] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: No bootstrap required, opened a new log
I20240501 13:59:28.714282 24184 ts_tablet_manager.cc:1367] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 13:59:28.718833 24182 raft_consensus.cc:348] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.719259 24182 raft_consensus.cc:374] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.719358 24182 raft_consensus.cc:729] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef279d2a404ab0a89d93cc7d8eaab6, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.720038 24182 consensus_queue.cc:260] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.720096 24183 raft_consensus.cc:348] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.720407 24183 raft_consensus.cc:374] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.720517 24183 raft_consensus.cc:729] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a507c6ded2492099ac01fba43a24be, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.721300 24183 consensus_queue.cc:260] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.721895 24182 ts_tablet_manager.cc:1398] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20240501 13:59:28.722424 23775 tablet.cc:2369] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:28.723026 24164 heartbeater.cc:503] Master 127.22.5.254:38759 was elected leader, sending a full tablet report...
I20240501 13:59:28.722774 24184 raft_consensus.cc:348] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.723570 24184 raft_consensus.cc:374] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.723670 24184 raft_consensus.cc:729] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6629f605fa14313ac9d5fe78ce15563, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.724247 24184 consensus_queue.cc:260] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.724424 24183 ts_tablet_manager.cc:1398] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20240501 13:59:28.726394 24184 ts_tablet_manager.cc:1398] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
W20240501 13:59:28.775194 23644 tablet.cc:2369] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:28.854003 24165 tablet.cc:2369] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:28.877431 24190 raft_consensus.cc:482] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:28.877760 24190 raft_consensus.cc:504] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.880249 24190 leader_election.cc:288] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bef279d2a404ab0a89d93cc7d8eaab6 (127.22.5.194:41701), 12a507c6ded2492099ac01fba43a24be (127.22.5.197:41625)
I20240501 13:59:28.889037 24119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3436c0b6834bfabbe2d7f7a0a1b99b" candidate_uuid: "e6629f605fa14313ac9d5fe78ce15563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a507c6ded2492099ac01fba43a24be" is_pre_election: true
I20240501 13:59:28.889257 23729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3436c0b6834bfabbe2d7f7a0a1b99b" candidate_uuid: "e6629f605fa14313ac9d5fe78ce15563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" is_pre_election: true
I20240501 13:59:28.889602 24119 raft_consensus.cc:2454] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6629f605fa14313ac9d5fe78ce15563 in term 0.
I20240501 13:59:28.889886 23729 raft_consensus.cc:2454] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6629f605fa14313ac9d5fe78ce15563 in term 0.
I20240501 13:59:28.890225 23531 leader_election.cc:302] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a507c6ded2492099ac01fba43a24be, e6629f605fa14313ac9d5fe78ce15563; no voters:
I20240501 13:59:28.890631 24190 raft_consensus.cc:2781] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:28.890790 24190 raft_consensus.cc:482] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:28.890897 24190 raft_consensus.cc:3037] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.892868 24190 raft_consensus.cc:504] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.893456 24190 leader_election.cc:288] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [CANDIDATE]: Term 1 election: Requested vote from peers 7bef279d2a404ab0a89d93cc7d8eaab6 (127.22.5.194:41701), 12a507c6ded2492099ac01fba43a24be (127.22.5.197:41625)
I20240501 13:59:28.893940 24119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3436c0b6834bfabbe2d7f7a0a1b99b" candidate_uuid: "e6629f605fa14313ac9d5fe78ce15563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a507c6ded2492099ac01fba43a24be"
I20240501 13:59:28.893985 23729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3436c0b6834bfabbe2d7f7a0a1b99b" candidate_uuid: "e6629f605fa14313ac9d5fe78ce15563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6"
I20240501 13:59:28.894117 24119 raft_consensus.cc:3037] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.894229 23729 raft_consensus.cc:3037] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.896183 24119 raft_consensus.cc:2454] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 12a507c6ded2492099ac01fba43a24be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6629f605fa14313ac9d5fe78ce15563 in term 1.
I20240501 13:59:28.896543 23531 leader_election.cc:302] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a507c6ded2492099ac01fba43a24be, e6629f605fa14313ac9d5fe78ce15563; no voters:
I20240501 13:59:28.896795 24190 raft_consensus.cc:2781] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.897858 23729 raft_consensus.cc:2454] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P 7bef279d2a404ab0a89d93cc7d8eaab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6629f605fa14313ac9d5fe78ce15563 in term 1.
I20240501 13:59:28.898332 24190 raft_consensus.cc:686] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [term 1 LEADER]: Becoming Leader. State: Replica: e6629f605fa14313ac9d5fe78ce15563, State: Running, Role: LEADER
I20240501 13:59:28.903540 24190 consensus_queue.cc:237] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } }
I20240501 13:59:28.909458 23456 catalog_manager.cc:5484] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6629f605fa14313ac9d5fe78ce15563 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "e6629f605fa14313ac9d5fe78ce15563" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6629f605fa14313ac9d5fe78ce15563" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:28.971176 22551 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:28.974339 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e6629f605fa14313ac9d5fe78ce15563 to finish bootstrapping
I20240501 13:59:28.984719 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7bef279d2a404ab0a89d93cc7d8eaab6 to finish bootstrapping
I20240501 13:59:28.994171 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1dd3358187ed4fb687306945e9bbeed2 to finish bootstrapping
I20240501 13:59:29.005990 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3747dabbc716432fb2b1939c585f4314 to finish bootstrapping
I20240501 13:59:29.016479 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12a507c6ded2492099ac01fba43a24be to finish bootstrapping
I20240501 13:59:29.158129 24034 heartbeater.cc:503] Master 127.22.5.254:38759 was elected leader, sending a full tablet report...
I20240501 13:59:29.410008 24210 consensus_queue.cc:1035] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bef279d2a404ab0a89d93cc7d8eaab6" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:29.424715 24210 consensus_queue.cc:1035] T 5b3436c0b6834bfabbe2d7f7a0a1b99b P e6629f605fa14313ac9d5fe78ce15563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a507c6ded2492099ac01fba43a24be" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:29.501614 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.508191 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.510609 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.513916 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.524818 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:29.675112 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.685880 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.688127 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.692059 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.703549 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:29.866904 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.870658 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.873536 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:29.881171 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:29.881163 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.494726 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.503468 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.508347 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.515772 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.531111 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:30.674036 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.688614 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.691310 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.691529 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.702694 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:30.858937 23839 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.861591 24099 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.873853 23709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.893730 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:30.902626 23578 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:31.036813 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 23516
I20240501 13:59:31.038178 23638 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.135936 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 23516
I20240501 13:59:31.160380 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 23647
I20240501 13:59:31.161460 23769 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.268530 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 23647
I20240501 13:59:31.287362 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 23777
I20240501 13:59:31.288396 23898 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.385248 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 23777
I20240501 13:59:31.403887 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 23907
I20240501 13:59:31.404946 24029 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.498507 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 23907
I20240501 13:59:31.516716 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24037
I20240501 13:59:31.517949 24159 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.614183 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24037
I20240501 13:59:31.632690 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 23426
I20240501 13:59:31.633862 23487 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:31.750928 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 23426
2024-05-01T13:59:31Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (6051 ms)
[----------] 1 test from RebalanceStartSafetyTest (6051 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240501 13:59:31.795701 22551 test_util.cc:268] Using random seed: -447604030
I20240501 13:59:31.797592 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:31.797693 22551 ts_itest-base.cc:113] --------------
I20240501 13:59:31.797735 22551 ts_itest-base.cc:114] 7 tablet servers
I20240501 13:59:31.797828 22551 ts_itest-base.cc:115] 5 replicas per TS
I20240501 13:59:31.797873 22551 ts_itest-base.cc:116] --------------
2024-05-01T13:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:31Z Disabled control of system clock
I20240501 13:59:31.823343 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:40379
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:40379
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 13:59:32.153159 24424 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 13:59:32.153538 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:32.153656 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:32.153788 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:32.161626 24424 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:32.161760 24424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:32.161875 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:32.161942 24424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:32.162041 24424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:32.171555 24424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:40379
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:32.172971 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:32.174852 24424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:32.186329 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:32.186830 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:32.187228 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:32.187920 24424 server_base.cc:1034] running on GCE node
I20240501 13:59:32.188685 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:32.190150 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:32.191380 24424 hybrid_clock.cc:648] HybridClock initialized: now 1714571972191305 us; error 38 us; skew 500 ppm
I20240501 13:59:32.191701 24424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:32.194068 24424 webserver.cc:460] Webserver started at http://127.22.5.254:39817/ using document root <none> and password file <none>
I20240501 13:59:32.194855 24424 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:32.194989 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:32.195328 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:32.198096 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5c4f17b3dc294f979e57abb381a3996f"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.198896 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5c4f17b3dc294f979e57abb381a3996f"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.204038 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:59:32.207311 24438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:32.208806 24424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 13:59:32.208974 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5c4f17b3dc294f979e57abb381a3996f"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.209205 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:32.227447 24424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:32.228241 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:32.228513 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:32.246112 24424 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:40379
I20240501 13:59:32.246140 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:40379 every 8 connection(s)
I20240501 13:59:32.247483 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:32.251204 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:32.256281 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24424
I20240501 13:59:32.256541 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:32.261792 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f: Bootstrap starting.
I20240501 13:59:32.264842 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:32.265898 24490 log.cc:824] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:32.268532 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f: No bootstrap required, opened a new log
I20240501 13:59:32.274993 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } }
I20240501 13:59:32.275347 24490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:32.275451 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4f17b3dc294f979e57abb381a3996f, State: Initialized, Role: FOLLOWER
I20240501 13:59:32.276005 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } }
I20240501 13:59:32.276206 24490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:32.276322 24490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:32.276470 24490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:32.278856 24490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } }
I20240501 13:59:32.279320 24490 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4f17b3dc294f979e57abb381a3996f; no voters:
I20240501 13:59:32.279881 24490 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:32.280015 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:32.280615 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [term 1 LEADER]: Becoming Leader. State: Replica: 5c4f17b3dc294f979e57abb381a3996f, State: Running, Role: LEADER
I20240501 13:59:32.281298 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } }
I20240501 13:59:32.282533 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:32.283919 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4f17b3dc294f979e57abb381a3996f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } } }
I20240501 13:59:32.283942 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4f17b3dc294f979e57abb381a3996f. Latest consensus state: current_term: 1 leader_uuid: "5c4f17b3dc294f979e57abb381a3996f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4f17b3dc294f979e57abb381a3996f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 40379 } } }
I20240501 13:59:32.284247 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:32.284250 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:32.286064 24500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:32.292189 24500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:32.301468 24500 catalog_manager.cc:1347] Generated new cluster ID: 70da445d34134f0a8c82e9329af6f34f
I20240501 13:59:32.301585 24500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:32.342700 24500 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:32.343909 24500 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:32.351998 24500 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5c4f17b3dc294f979e57abb381a3996f: Generated new TSK 0
I20240501 13:59:32.352887 24500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:32.371414 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:32.700350 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:32.700711 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:32.700877 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:32.708659 24514 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:32.708832 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:32.709007 24514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 13:59:32.718556 24514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:32.720028 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:32.721958 24514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:32.734534 24519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:32.735069 24520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:32.735498 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:32.736069 24514 server_base.cc:1034] running on GCE node
I20240501 13:59:32.736718 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:32.737787 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:32.739029 24514 hybrid_clock.cc:648] HybridClock initialized: now 1714571972738974 us; error 42 us; skew 500 ppm
I20240501 13:59:32.739353 24514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:32.745266 24514 webserver.cc:460] Webserver started at http://127.22.5.193:40781/ using document root <none> and password file <none>
I20240501 13:59:32.746042 24514 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:32.746141 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:32.746423 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:32.749087 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "283529d70ba14547b9440a16350b629e"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.749976 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "283529d70ba14547b9440a16350b629e"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.755259 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:32.758991 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:32.760334 24514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:32.760515 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "283529d70ba14547b9440a16350b629e"
format_stamp: "Formatted at 2024-05-01 13:59:32 on dist-test-slave-5jzj"
I20240501 13:59:32.760717 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:32.780138 24514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:32.781035 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:32.781335 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:32.782788 24514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:32.784286 24514 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:32.784381 24514 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:32.784503 24514 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:32.784575 24514 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:32.815850 24514 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:35547
I20240501 13:59:32.815898 24640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:35547 every 8 connection(s)
I20240501 13:59:32.817214 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:32.820320 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24514
I20240501 13:59:32.820528 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:32.823917 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:32.831517 24641 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:32.831881 24641 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:32.832988 24641 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:32.835301 24455 ts_manager.cc:194] Registered new tserver with Master: 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 13:59:32.836858 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:57719
W20240501 13:59:33.161056 24645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:33.161470 24645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:33.161610 24645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:33.169478 24645 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:33.169642 24645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:33.169811 24645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 13:59:33.180896 24645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:33.182339 24645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:33.184268 24645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:33.196432 24650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.196954 24651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.197387 24653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:33.197986 24645 server_base.cc:1034] running on GCE node
I20240501 13:59:33.198632 24645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:33.199658 24645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:33.200892 24645 hybrid_clock.cc:648] HybridClock initialized: now 1714571973200836 us; error 45 us; skew 500 ppm
I20240501 13:59:33.201289 24645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:33.203668 24645 webserver.cc:460] Webserver started at http://127.22.5.194:37025/ using document root <none> and password file <none>
I20240501 13:59:33.204409 24645 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:33.204517 24645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:33.204845 24645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:33.207793 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "909bf3bda47b4c7ab369f460f5e96973"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.208662 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "909bf3bda47b4c7ab369f460f5e96973"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.214109 24645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:33.217732 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.219148 24645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 13:59:33.219311 24645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "909bf3bda47b4c7ab369f460f5e96973"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.219516 24645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:33.240761 24645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:33.241729 24645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:33.242012 24645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:33.243000 24645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:33.244660 24645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:33.244763 24645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.244889 24645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:33.244980 24645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.277045 24645 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:45987
I20240501 13:59:33.277089 24771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:45987 every 8 connection(s)
I20240501 13:59:33.278376 24645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:33.284384 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24645
I20240501 13:59:33.284569 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:33.287762 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:33.291615 24772 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:33.291941 24772 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:33.293058 24772 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:33.294724 24455 ts_manager.cc:194] Registered new tserver with Master: 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 13:59:33.295862 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:46669
W20240501 13:59:33.621080 24775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:33.621445 24775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:33.621549 24775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:33.629278 24775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:33.629413 24775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:33.629544 24775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 13:59:33.638774 24775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:33.640116 24775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:33.641927 24775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:33.654486 24781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.654958 24780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:33.655373 24783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:33.656025 24775 server_base.cc:1034] running on GCE node
I20240501 13:59:33.656668 24775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:33.657793 24775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:33.659003 24775 hybrid_clock.cc:648] HybridClock initialized: now 1714571973658956 us; error 43 us; skew 500 ppm
I20240501 13:59:33.659327 24775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:33.661695 24775 webserver.cc:460] Webserver started at http://127.22.5.195:35607/ using document root <none> and password file <none>
I20240501 13:59:33.662678 24775 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:33.662827 24775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:33.663189 24775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:33.666229 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f74cb76209d04397a8a206f86e91ac5b"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.667160 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f74cb76209d04397a8a206f86e91ac5b"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.672897 24775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:59:33.676792 24789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.678323 24775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:33.678496 24775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f74cb76209d04397a8a206f86e91ac5b"
format_stamp: "Formatted at 2024-05-01 13:59:33 on dist-test-slave-5jzj"
I20240501 13:59:33.678707 24775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:33.700738 24775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:33.701678 24775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:33.701967 24775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:33.702931 24775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:33.704463 24775 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:33.704564 24775 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.704682 24775 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:33.704761 24775 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:33.736560 24775 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:37091
I20240501 13:59:33.736615 24901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:37091 every 8 connection(s)
I20240501 13:59:33.737911 24775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:33.738538 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24775
I20240501 13:59:33.738706 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:33.742282 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:33.752408 24902 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:33.752732 24902 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:33.753597 24902 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:33.755108 24455 ts_manager.cc:194] Registered new tserver with Master: f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 13:59:33.755894 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:55909
I20240501 13:59:33.840615 24641 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
W20240501 13:59:34.078430 24905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:34.078791 24905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:34.078945 24905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:34.086701 24905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:34.086860 24905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:34.087023 24905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 13:59:34.096285 24905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:34.097749 24905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:34.099534 24905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:34.111990 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.112465 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.112901 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:34.113487 24905 server_base.cc:1034] running on GCE node
I20240501 13:59:34.114168 24905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:34.115353 24905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:34.116597 24905 hybrid_clock.cc:648] HybridClock initialized: now 1714571974116503 us; error 89 us; skew 500 ppm
I20240501 13:59:34.116943 24905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:34.119308 24905 webserver.cc:460] Webserver started at http://127.22.5.196:37821/ using document root <none> and password file <none>
I20240501 13:59:34.120074 24905 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:34.120191 24905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:34.120508 24905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:34.123201 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.124011 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.129424 24905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:34.133040 24919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.134517 24905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:34.134685 24905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.134888 24905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:34.158686 24905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:34.159574 24905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:34.159857 24905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:34.160812 24905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:34.162357 24905 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:34.162456 24905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.162577 24905 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:34.162648 24905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.194367 24905 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:43227
I20240501 13:59:34.194413 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:43227 every 8 connection(s)
I20240501 13:59:34.195799 24905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:34.202380 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 24905
I20240501 13:59:34.202579 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:34.205544 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.197:0
--local_ip_for_outbound_sockets=127.22.5.197
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:34.208885 25032 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:34.209259 25032 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:34.210163 25032 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:34.211709 24455 ts_manager.cc:194] Registered new tserver with Master: 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:34.212911 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:38363
I20240501 13:59:34.299003 24772 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
W20240501 13:59:34.542789 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:34.543085 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:34.543231 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:34.550648 25035 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:34.550765 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:34.550921 25035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.197
I20240501 13:59:34.559695 25035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.197
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:34.561076 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:34.562870 25035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:34.575196 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.575713 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:34.576227 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:34.578950 25035 server_base.cc:1034] running on GCE node
I20240501 13:59:34.579562 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:34.580543 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:34.581743 25035 hybrid_clock.cc:648] HybridClock initialized: now 1714571974581693 us; error 49 us; skew 500 ppm
I20240501 13:59:34.582055 25035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:34.584510 25035 webserver.cc:460] Webserver started at http://127.22.5.197:33649/ using document root <none> and password file <none>
I20240501 13:59:34.585273 25035 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:34.585388 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:34.585695 25035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:34.588482 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "491e21cc623d4879b09975ede207ab2d"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.589301 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "491e21cc623d4879b09975ede207ab2d"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.594476 25035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:34.597994 25049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.599336 25035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:34.599489 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "491e21cc623d4879b09975ede207ab2d"
format_stamp: "Formatted at 2024-05-01 13:59:34 on dist-test-slave-5jzj"
I20240501 13:59:34.599687 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:34.622110 25035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:34.623001 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:34.623281 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:34.624285 25035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:34.625929 25035 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:34.626036 25035 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.626159 25035 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:34.626235 25035 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:34.657370 25035 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.197:35069
I20240501 13:59:34.657415 25161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.197:35069 every 8 connection(s)
I20240501 13:59:34.658699 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:34.665009 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 25035
I20240501 13:59:34.665220 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:34.668759 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.198:0
--local_ip_for_outbound_sockets=127.22.5.198
--webserver_interface=127.22.5.198
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:34.672540 25162 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:34.672873 25162 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:34.673873 25162 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:34.675240 24455 ts_manager.cc:194] Registered new tserver with Master: 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:34.675992 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.197:55001
I20240501 13:59:34.758478 24902 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
W20240501 13:59:34.995070 25165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:34.995361 25165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:34.995493 25165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:35.003198 25165 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:35.003322 25165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:35.003454 25165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.198
I20240501 13:59:35.012539 25165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.5.198
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.198
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:35.013983 25165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:35.015736 25165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:35.028828 25170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.029321 25171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.029716 25173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:35.030366 25165 server_base.cc:1034] running on GCE node
I20240501 13:59:35.030977 25165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:35.031946 25165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:35.033172 25165 hybrid_clock.cc:648] HybridClock initialized: now 1714571975033103 us; error 41 us; skew 500 ppm
I20240501 13:59:35.033497 25165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:35.035955 25165 webserver.cc:460] Webserver started at http://127.22.5.198:45601/ using document root <none> and password file <none>
I20240501 13:59:35.036707 25165 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:35.036828 25165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:35.037211 25165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:35.039943 25165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.040712 25165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.045883 25165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240501 13:59:35.049284 25179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.050762 25165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:35.050992 25165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.051218 25165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:35.069576 25165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:35.070422 25165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:35.070701 25165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:35.071686 25165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:35.073194 25165 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:35.073293 25165 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.073408 25165 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:35.073486 25165 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.103701 25165 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.198:32877
I20240501 13:59:35.103745 25291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.198:32877 every 8 connection(s)
I20240501 13:59:35.105197 25165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:35.107228 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 25165
I20240501 13:59:35.107410 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 13:59:35.110622 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.199:0
--local_ip_for_outbound_sockets=127.22.5.199
--webserver_interface=127.22.5.199
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:35.119148 25292 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:35.119499 25292 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:35.120568 25292 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:35.122193 24455 ts_manager.cc:194] Registered new tserver with Master: 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:35.123076 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.198:57533
I20240501 13:59:35.215672 25032 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
W20240501 13:59:35.450747 25295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:35.451069 25295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:35.451234 25295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:35.459010 25295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:35.459151 25295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:35.459312 25295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.199
I20240501 13:59:35.468449 25295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.5.199
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.199
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:35.469877 25295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:35.471673 25295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:35.484345 25303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.484712 25301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:35.485248 25300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:35.490370 25295 server_base.cc:1034] running on GCE node
I20240501 13:59:35.491026 25295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:35.492022 25295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:35.493230 25295 hybrid_clock.cc:648] HybridClock initialized: now 1714571975493172 us; error 62 us; skew 500 ppm
I20240501 13:59:35.493559 25295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:35.495908 25295 webserver.cc:460] Webserver started at http://127.22.5.199:39691/ using document root <none> and password file <none>
I20240501 13:59:35.496680 25295 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:35.496799 25295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:35.497175 25295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:35.499800 25295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2f6d2177839a452eba5e2fc39cb40881"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.500573 25295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2f6d2177839a452eba5e2fc39cb40881"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.505980 25295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 13:59:35.509533 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.511001 25295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 13:59:35.511170 25295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2f6d2177839a452eba5e2fc39cb40881"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:35.511374 25295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:35.536432 25295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:35.537371 25295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:35.537653 25295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:35.538712 25295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:35.540324 25295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:35.540424 25295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.540536 25295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:35.540613 25295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:35.572113 25295 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.199:35589
I20240501 13:59:35.572171 25421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.199:35589 every 8 connection(s)
I20240501 13:59:35.573459 25295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:35.580920 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 25295
I20240501 13:59:35.581158 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 13:59:35.586906 25422 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:35.587210 25422 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:35.588068 25422 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:35.589478 24455 ts_manager.cc:194] Registered new tserver with Master: 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 13:59:35.590252 24455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.199:53275
I20240501 13:59:35.597560 22551 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:35.613337 24455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:35.668932 25357 tablet_service.cc:1467] Processing CreateTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 (DEFAULT_TABLE table=TestTable [id=037bfbab72cb4186830bb25698a63714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:35.668933 24707 tablet_service.cc:1467] Processing CreateTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 (DEFAULT_TABLE table=TestTable [id=037bfbab72cb4186830bb25698a63714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:35.670572 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:35.670604 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:35.678318 25162 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
I20240501 13:59:35.677839 25097 tablet_service.cc:1467] Processing CreateTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 (DEFAULT_TABLE table=TestTable [id=037bfbab72cb4186830bb25698a63714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:35.679518 25097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:35.683251 24837 tablet_service.cc:1467] Processing CreateTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 (DEFAULT_TABLE table=TestTable [id=037bfbab72cb4186830bb25698a63714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:35.684170 25227 tablet_service.cc:1467] Processing CreateTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 (DEFAULT_TABLE table=TestTable [id=037bfbab72cb4186830bb25698a63714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:35.684792 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:35.685532 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:35.693818 25444 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:35.699409 25444 tablet_bootstrap.cc:654] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:35.700783 25444 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:35.702296 25445 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:35.704728 25446 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 13:59:35.706431 25445 tablet_bootstrap.cc:654] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:35.706604 25447 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:35.707804 25445 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:35.710455 25447 tablet_bootstrap.cc:654] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:35.715128 25444 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:35.715497 25444 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.011s
I20240501 13:59:35.716185 25446 tablet_bootstrap.cc:654] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:35.717231 25445 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:35.717589 25445 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240501 13:59:35.717932 25449 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:35.719987 25447 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:35.721149 25446 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:35.721452 25449 tablet_bootstrap.cc:654] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:35.722812 25449 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:35.725088 25444 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.725762 25444 raft_consensus.cc:374] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:35.725908 25444 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:35.726817 25444 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.726899 25445 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.728121 25445 raft_consensus.cc:374] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:35.728273 25445 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:35.730013 25446 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: No bootstrap required, opened a new log
I20240501 13:59:35.730011 25447 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:35.730326 25447 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240501 13:59:35.730325 25446 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240501 13:59:35.730602 25449 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:35.731019 25449 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.005s
I20240501 13:59:35.739836 25446 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.740422 25446 raft_consensus.cc:374] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:35.740551 25446 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: FOLLOWER
I20240501 13:59:35.741506 25446 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.746402 25449 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.747107 25449 raft_consensus.cc:374] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:35.747282 25449 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:35.748405 25449 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.749851 25446 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20240501 13:59:35.750226 25422 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
I20240501 13:59:35.753343 25445 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.754827 25447 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.755486 25444 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.040s user 0.023s sys 0.014s
I20240501 13:59:35.755764 25447 raft_consensus.cc:374] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:35.755884 25447 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:35.757473 25447 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.761365 25449 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.030s user 0.023s sys 0.003s
I20240501 13:59:35.761762 25445 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.044s user 0.013s sys 0.000s
I20240501 13:59:35.762835 25292 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
I20240501 13:59:35.763374 25447 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.033s user 0.013s sys 0.000s
W20240501 13:59:35.781756 24773 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:35.825052 25423 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:35.857259 25293 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 13:59:35.911259 25163 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:35.946381 25454 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:35.946673 25454 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.948827 25454 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:35.957625 25457 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:35.958146 25457 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.959501 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:35.960060 25247 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:35.961470 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 13:59:35.961941 25377 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:35.962159 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:35.962671 24857 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:35.962564 24663 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:35.963524 25454 raft_consensus.cc:2781] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:35.963747 25454 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:35.963860 25454 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:35.966157 25457 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 13:59:35.967792 25454 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.969149 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:35.969676 25117 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:35.971170 25454 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:35.975230 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881"
I20240501 13:59:35.975473 25377 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:35.978615 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:35.979146 24727 raft_consensus.cc:2379] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 491e21cc623d4879b09975ede207ab2d in current term 1: Already voted for candidate 909bf3bda47b4c7ab369f460f5e96973 in this term.
I20240501 13:59:35.980664 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:35.980908 25247 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:35.981217 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:35.981422 25117 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:35.981772 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:35.981966 25247 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:35.981967 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:35.982196 24857 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:35.984912 25117 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:35.984978 25247 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:35.985404 24857 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:35.985636 25377 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:35.986052 24663 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, 491e21cc623d4879b09975ede207ab2d, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:35.986490 25454 raft_consensus.cc:2781] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:35.986919 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:35.987056 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 13:59:35.987313 25454 raft_consensus.cc:686] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Becoming Leader. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Running, Role: LEADER
I20240501 13:59:35.987382 24857 raft_consensus.cc:2379] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 491e21cc623d4879b09975ede207ab2d in current term 1: Already voted for candidate 909bf3bda47b4c7ab369f460f5e96973 in this term.
I20240501 13:59:35.987524 25377 raft_consensus.cc:2379] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 491e21cc623d4879b09975ede207ab2d in current term 1: Already voted for candidate 909bf3bda47b4c7ab369f460f5e96973 in this term.
I20240501 13:59:35.987963 25454 consensus_queue.cc:237] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:35.988137 25053 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd; no voters: 2f6d2177839a452eba5e2fc39cb40881, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b
I20240501 13:59:35.988548 25457 raft_consensus.cc:2726] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 13:59:35.991202 24903 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:35.993438 24454 catalog_manager.cc:5484] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:36.048075 22551 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:36.050215 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 283529d70ba14547b9440a16350b629e to finish bootstrapping
I20240501 13:59:36.059319 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 909bf3bda47b4c7ab369f460f5e96973 to finish bootstrapping
I20240501 13:59:36.066368 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f74cb76209d04397a8a206f86e91ac5b to finish bootstrapping
I20240501 13:59:36.073328 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e03e0d09b974c7d8cf7f49652d4fd23 to finish bootstrapping
I20240501 13:59:36.081897 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 491e21cc623d4879b09975ede207ab2d to finish bootstrapping
I20240501 13:59:36.088994 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 84c5917a503d4a82bb2d9c0d5fdb04bd to finish bootstrapping
I20240501 13:59:36.095880 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2f6d2177839a452eba5e2fc39cb40881 to finish bootstrapping
I20240501 13:59:36.103094 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 25295
I20240501 13:59:36.104089 25417 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:36.196367 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 25295
I20240501 13:59:36.369289 25454 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:36.372713 24663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111)
W20240501 13:59:36.374169 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:36.390470 25454 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:36.392757 25454 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:38.768965 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:39.965781 24454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:39.977821 24707 tablet_service.cc:1467] Processing CreateTablet for tablet f64ab2203c9042989dbcab79bab164a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.978305 25097 tablet_service.cc:1467] Processing CreateTablet for tablet f64ab2203c9042989dbcab79bab164a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.978529 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:39.978875 25097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:39.984515 24837 tablet_service.cc:1467] Processing CreateTablet for tablet f64ab2203c9042989dbcab79bab164a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.984764 25227 tablet_service.cc:1467] Processing CreateTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.985023 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:39.985196 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:39.994249 24706 tablet_service.cc:1467] Processing CreateTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.994904 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:39.997020 24836 tablet_service.cc:1467] Processing CreateTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:39.997481 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.002701 25096 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1344fe8e4742e5914f49ffcaba21f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.003196 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1344fe8e4742e5914f49ffcaba21f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.003386 25226 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1344fe8e4742e5914f49ffcaba21f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.003888 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1344fe8e4742e5914f49ffcaba21f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.005654 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1344fe8e4742e5914f49ffcaba21f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.006202 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1344fe8e4742e5914f49ffcaba21f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.019611 25500 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:40.020535 25502 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:40.022341 25500 tablet_bootstrap.cc:654] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.022990 25501 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:40.023533 25503 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:40.024768 25502 tablet_bootstrap.cc:654] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.026984 25503 tablet_bootstrap.cc:654] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.028275 25501 tablet_bootstrap.cc:654] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.030128 25500 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:40.030287 25500 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 13:59:40.031236 25500 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.031520 25500 raft_consensus.cc:374] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.031643 25500 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.031069 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1344fe8e4742e5914f49ffcaba21f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.031931 25500 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.032917 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1344fe8e4742e5914f49ffcaba21f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.033255 24576 tablet_service.cc:1467] Processing CreateTablet for tablet f64ab2203c9042989dbcab79bab164a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.033800 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.040292 24575 tablet_service.cc:1467] Processing CreateTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.041873 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.042979 25503 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:40.043092 25500 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20240501 13:59:40.043243 25503 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240501 13:59:40.043792 25500 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:40.044441 25503 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.044783 25503 raft_consensus.cc:374] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.044943 25503 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.045511 25500 tablet_bootstrap.cc:654] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.050068 25501 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:40.050314 25501 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20240501 13:59:40.051383 25501 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.051687 25501 raft_consensus.cc:374] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.051043 24965 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1344fe8e4742e5914f49ffcaba21f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.051795 25501 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.052070 25501 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.045311 25503 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.052914 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1344fe8e4742e5914f49ffcaba21f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.053608 24966 tablet_service.cc:1467] Processing CreateTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.054328 24967 tablet_service.cc:1467] Processing CreateTablet for tablet f64ab2203c9042989dbcab79bab164a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:40.055629 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.056937 25502 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:40.057157 25502 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20240501 13:59:40.058444 25502 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.058763 25502 raft_consensus.cc:374] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.058877 25502 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.061511 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:40.067432 25500 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:40.067658 25500 ts_tablet_manager.cc:1367] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20240501 13:59:40.068483 25500 raft_consensus.cc:348] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.068795 25500 raft_consensus.cc:374] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.068872 25500 raft_consensus.cc:729] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.069156 25500 consensus_queue.cc:260] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.069553 25502 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.070807 25502 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240501 13:59:40.071280 25502 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:40.073573 25502 tablet_bootstrap.cc:654] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.073998 25512 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:40.078815 25512 tablet_bootstrap.cc:654] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.079635 25502 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:40.079887 25502 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 13:59:40.080329 25512 log.cc:824] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:40.080739 25502 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.081229 25500 ts_tablet_manager.cc:1398] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240501 13:59:40.081231 25502 raft_consensus.cc:374] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.081444 25502 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.081854 25502 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.082805 25502 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:40.083225 25502 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:40.085556 25502 tablet_bootstrap.cc:654] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.091252 25502 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:40.091442 25502 ts_tablet_manager.cc:1367] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 13:59:40.092377 25512 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:40.092244 25502 raft_consensus.cc:348] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.092563 25502 raft_consensus.cc:374] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.092660 25502 raft_consensus.cc:729] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.093009 25512 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.011s
I20240501 13:59:40.092954 25502 consensus_queue.cc:260] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.093575 25502 ts_tablet_manager.cc:1398] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:40.094503 25516 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:40.098878 25516 tablet_bootstrap.cc:654] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.100456 25516 log.cc:824] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:40.103461 25512 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.104836 25512 raft_consensus.cc:374] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.106649 25518 raft_consensus.cc:482] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:40.106923 25512 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.107241 25518 raft_consensus.cc:504] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.108378 25512 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.110551 25516 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:40.111039 25516 ts_tablet_manager.cc:1367] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240501 13:59:40.117481 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:40.117780 24727 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:40.119622 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:40.119910 24857 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:40.120941 25053 leader_election.cc:302] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:40.121750 25518 leader_election.cc:288] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 13:59:40.122365 25518 raft_consensus.cc:2781] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:40.122130 25516 raft_consensus.cc:348] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.122843 25516 raft_consensus.cc:374] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.122910 25518 raft_consensus.cc:482] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:40.123021 25518 raft_consensus.cc:3037] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.123049 25516 raft_consensus.cc:729] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.124186 25516 consensus_queue.cc:260] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.126122 25501 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.076s user 0.008s sys 0.023s
I20240501 13:59:40.126437 25501 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:40.128187 25501 tablet_bootstrap.cc:654] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.129451 25503 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.086s user 0.013s sys 0.018s
I20240501 13:59:40.129846 25503 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:40.131716 25503 tablet_bootstrap.cc:654] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.135332 25512 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.042s user 0.014s sys 0.002s
I20240501 13:59:40.139225 25512 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:40.142223 25516 ts_tablet_manager.cc:1398] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.031s user 0.020s sys 0.000s
I20240501 13:59:40.142832 25516 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:40.144580 25516 tablet_bootstrap.cc:654] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.146133 25501 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:40.147295 25501 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20240501 13:59:40.147326 25512 tablet_bootstrap.cc:654] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.148372 25501 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.148751 25501 raft_consensus.cc:374] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.148893 25501 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.149299 25501 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.149808 25503 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:40.150002 25503 ts_tablet_manager.cc:1367] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.009s
I20240501 13:59:40.150192 25501 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:40.151010 25503 raft_consensus.cc:348] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.152917 25503 raft_consensus.cc:374] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.153052 25503 raft_consensus.cc:729] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.153595 25503 consensus_queue.cc:260] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.154459 25503 ts_tablet_manager.cc:1398] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:40.155305 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:40.155568 25516 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:40.155738 25516 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 13:59:40.155979 24596 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:40.156356 25512 tablet_bootstrap.cc:492] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:40.156508 25512 ts_tablet_manager.cc:1367] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240501 13:59:40.156325 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:40.157161 25516 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.157397 25516 raft_consensus.cc:374] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.157538 25516 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
W20240501 13:59:40.157506 25050 leader_election.cc:341] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:40.157521 25512 raft_consensus.cc:348] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.157810 25512 raft_consensus.cc:374] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.157789 25516 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.157954 25512 raft_consensus.cc:729] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.158191 25512 consensus_queue.cc:260] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.158396 25516 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:40.158663 25516 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:40.158869 25512 ts_tablet_manager.cc:1398] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:40.159118 25512 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:40.160362 25516 tablet_bootstrap.cc:654] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.160856 25512 tablet_bootstrap.cc:654] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:40.165745 25516 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:40.165908 25516 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:59:40.165956 25512 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:40.166126 25512 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240501 13:59:40.166560 25516 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.166755 25516 raft_consensus.cc:374] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.166853 25516 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.166762 25512 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.167081 25512 raft_consensus.cc:374] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:40.167181 25512 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:40.167081 25516 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.167621 25516 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:40.167467 25512 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.168686 25512 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:40.194907 25518 raft_consensus.cc:504] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.196256 25518 leader_election.cc:288] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 13:59:40.197031 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:40.197271 24727 raft_consensus.cc:3037] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.199234 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:40.199527 24857 raft_consensus.cc:3037] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.199882 25480 raft_consensus.cc:482] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:40.200057 25480 raft_consensus.cc:504] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.200220 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:40.200484 24987 raft_consensus.cc:3037] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.201347 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64ab2203c9042989dbcab79bab164a6" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:40.201567 24596 raft_consensus.cc:3037] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:59:40.201784 25033 tablet.cc:2369] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:40.203804 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:40.204043 25247 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.204313 25480 leader_election.cc:288] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 13:59:40.205385 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:40.205730 24856 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.207214 24663 leader_election.cc:302] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:40.207942 25480 raft_consensus.cc:2781] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:40.208061 25480 raft_consensus.cc:482] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:40.208153 25480 raft_consensus.cc:3037] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.211827 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:40.212113 24595 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.212414 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:40.212795 24986 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.243409 25530 raft_consensus.cc:482] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:40.243633 25530 raft_consensus.cc:504] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.244576 25530 leader_election.cc:288] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:40.244997 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:40.245184 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:40.245184 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:40.245265 24595 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.245250 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:40.245456 25117 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.245457 25247 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.245489 24986 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:40.245849 24660 leader_election.cc:302] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:40.246155 25530 raft_consensus.cc:2781] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:40.246279 25530 raft_consensus.cc:482] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:40.246368 25530 raft_consensus.cc:3037] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.294694 24727 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:40.294680 24987 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:40.294704 24596 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:40.295420 25052 leader_election.cc:302] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:40.295445 24857 raft_consensus.cc:2454] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:40.295841 25518 raft_consensus.cc:2781] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:40.296020 25518 raft_consensus.cc:686] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Becoming Leader. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Running, Role: LEADER
I20240501 13:59:40.296602 25518 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.298485 25480 raft_consensus.cc:504] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.299609 25480 leader_election.cc:288] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 13:59:40.300307 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:40.300359 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:40.300560 24596 raft_consensus.cc:3037] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.300590 24987 raft_consensus.cc:3037] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.300653 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:40.300850 24857 raft_consensus.cc:3037] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.301774 24455 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 491e21cc623d4879b09975ede207ab2d (127.22.5.197). New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:40.303362 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:40.303620 25247 raft_consensus.cc:3037] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:59:40.324519 24642 tablet.cc:2369] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:40.389910 24596 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.389936 24857 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.389952 25530 raft_consensus.cc:504] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.390606 25247 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.391034 25530 leader_election.cc:288] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:40.391382 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:40.391474 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:40.391561 24663 leader_election.cc:302] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:40.391613 24596 raft_consensus.cc:3037] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.391700 24986 raft_consensus.cc:3037] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.391906 25530 raft_consensus.cc:2781] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:40.392027 25530 raft_consensus.cc:686] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Becoming Leader. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Running, Role: LEADER
I20240501 13:59:40.392745 24987 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.392352 25530 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } }
I20240501 13:59:40.394439 24596 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.395889 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:40.396133 25117 raft_consensus.cc:3037] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.396591 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:40.396811 25247 raft_consensus.cc:3037] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:40.397265 24455 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:40.489691 25247 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.489675 25117 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.489789 24986 raft_consensus.cc:2454] T 2f1344fe8e4742e5914f49ffcaba21f8 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:40.490098 24663 leader_election.cc:302] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:40.490491 25530 raft_consensus.cc:2781] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:40.490675 25530 raft_consensus.cc:686] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Becoming Leader. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Running, Role: LEADER
I20240501 13:59:40.490931 25530 consensus_queue.cc:237] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:40.494022 24455 catalog_manager.cc:5484] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:40.724220 25518 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.844496 25518 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.847748 25530 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.858047 25518 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.871992 25518 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.883710 25530 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.892735 25480 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.938704 25553 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.952461 25553 consensus_queue.cc:1035] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:40.966179 25535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:40.966543 25535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:40.974046 25535 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:40.985929 25553 consensus_queue.cc:1035] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.990900 25554 consensus_queue.cc:1035] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:40.994024 25553 consensus_queue.cc:1035] T 2f1344fe8e4742e5914f49ffcaba21f8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
W20240501 13:59:41.388160 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 52967.4us
W20240501 13:59:41.420663 24663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 13:59:41.447510 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
INSERT report
rows total: 2000
time total: 416.579 ms
time per row: 0.20829 ms
I20240501 13:59:41.554997 24448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:41.564888 24965 tablet_service.cc:1467] Processing CreateTablet for tablet 9762584f4ce94ab68a52bc1f446e569c (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.564996 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 9762584f4ce94ab68a52bc1f446e569c (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.565402 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.565402 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.565435 25227 tablet_service.cc:1467] Processing CreateTablet for tablet 9762584f4ce94ab68a52bc1f446e569c (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.565879 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.570691 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.571156 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.571436 25226 tablet_service.cc:1467] Processing CreateTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.571573 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 9762584f4ce94ab68a52bc1f446e569c (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.571841 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.572046 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.570742 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.574757 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.576864 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.577360 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.581652 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.582130 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.585757 25620 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:41.586624 25097 tablet_service.cc:1467] Processing CreateTablet for tablet 9762584f4ce94ab68a52bc1f446e569c (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.586817 25096 tablet_service.cc:1467] Processing CreateTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.587076 25097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.586952 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.587462 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.588025 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.588425 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.594023 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.594146 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.595180 25620 tablet_bootstrap.cc:654] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.595260 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.597605 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:41.598094 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.598608 25622 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:41.602250 25622 tablet_bootstrap.cc:654] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.603564 25620 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:41.603741 25620 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240501 13:59:41.605953 25623 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:41.607861 25623 tablet_bootstrap.cc:654] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.609711 25620 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.610002 25620 raft_consensus.cc:374] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.610109 25620 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.610419 25620 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.612083 25623 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:41.612251 25623 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 13:59:41.613094 25623 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.613351 25623 raft_consensus.cc:374] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.613461 25623 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.615056 25623 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.616271 25623 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 13:59:41.616590 25623 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:41.618290 25623 tablet_bootstrap.cc:654] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.621281 25630 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:41.625279 25630 tablet_bootstrap.cc:654] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.626680 25628 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:41.629030 25628 tablet_bootstrap.cc:654] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.630789 25621 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:41.631227 25620 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.027s user 0.002s sys 0.010s
I20240501 13:59:41.631572 25620 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:41.633311 25620 tablet_bootstrap.cc:654] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.647706 25621 tablet_bootstrap.cc:654] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.648016 25620 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:41.648216 25620 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.007s
I20240501 13:59:41.649158 25620 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.649503 25620 raft_consensus.cc:374] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.649596 25620 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.649914 25620 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.650976 25622 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:41.651259 25622 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.027s
I20240501 13:59:41.652345 25620 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 13:59:41.652670 25620 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:41.652411 25622 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.652993 25622 raft_consensus.cc:374] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.653179 25622 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.653548 25622 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.655975 25623 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:41.656257 25623 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.010s
I20240501 13:59:41.656787 25621 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:41.656980 25621 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240501 13:59:41.657296 25623 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.657735 25623 raft_consensus.cc:374] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.657817 25623 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.658097 25623 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.658731 25623 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:41.658952 25620 tablet_bootstrap.cc:654] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.660121 25623 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:41.659823 25628 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:41.660437 25628 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 13:59:41.661891 25623 tablet_bootstrap.cc:654] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.668488 25628 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.668849 25628 raft_consensus.cc:374] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.668957 25628 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.669312 25628 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.669741 25622 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.018s user 0.001s sys 0.005s
I20240501 13:59:41.670174 25622 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:41.671797 25622 tablet_bootstrap.cc:654] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.672279 25623 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:41.672446 25623 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240501 13:59:41.673357 25623 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.673560 25630 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:41.673643 25623 raft_consensus.cc:374] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.673714 25630 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.031s
I20240501 13:59:41.673748 25623 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.674342 25623 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.675069 25623 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:41.675419 25621 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.676394 25620 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:41.676589 25620 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.001s
I20240501 13:59:41.676908 25621 raft_consensus.cc:374] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.677033 25621 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.677418 25620 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.677706 25620 raft_consensus.cc:374] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.677806 25620 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.674547 25630 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.678041 25630 raft_consensus.cc:374] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.678092 25620 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.679325 25620 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:41.679598 25621 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.678153 25630 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.680382 25630 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.680860 25622 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:41.681236 25630 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 13:59:41.681699 25628 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 13:59:41.682104 25628 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:41.684080 25622 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240501 13:59:41.684716 25628 tablet_bootstrap.cc:654] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.685084 25622 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.685474 25622 raft_consensus.cc:374] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.685593 25622 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.685987 25622 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.686619 25622 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:41.687014 25622 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:41.688701 25622 tablet_bootstrap.cc:654] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.690236 25621 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.033s user 0.007s sys 0.004s
I20240501 13:59:41.691220 25621 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:41.691991 25628 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:41.692140 25622 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:41.692164 25628 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240501 13:59:41.692262 25622 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:41.692953 25622 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.692935 25628 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.693217 25622 raft_consensus.cc:374] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.693233 25628 raft_consensus.cc:374] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.693305 25628 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.693308 25622 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.693606 25628 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.693825 25621 tablet_bootstrap.cc:654] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.695266 25622 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.695955 25622 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:41.698086 25628 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 13:59:41.700421 25628 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:41.702348 25621 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:41.702510 25621 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 13:59:41.702538 25628 tablet_bootstrap.cc:654] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:41.703202 25621 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.703455 25621 raft_consensus.cc:374] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.703521 25621 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.703801 25621 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.704355 25621 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 13:59:41.705915 25628 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:41.706079 25628 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 13:59:41.706707 25628 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.706915 25628 raft_consensus.cc:374] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:41.707024 25628 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.707274 25628 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.708386 25546 raft_consensus.cc:482] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:41.708546 25546 raft_consensus.cc:504] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.709512 25546 leader_election.cc:288] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:41.709803 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:41.709954 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:41.710057 24857 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:41.710166 25247 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:41.710676 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:41.710901 24596 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:41.711282 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:41.711354 25050 leader_election.cc:302] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd; no voters:
I20240501 13:59:41.711508 24986 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 0.
I20240501 13:59:41.711632 25546 raft_consensus.cc:2781] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:41.711750 25546 raft_consensus.cc:482] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:41.711835 25546 raft_consensus.cc:3037] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.712638 25628 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240501 13:59:41.738595 25546 raft_consensus.cc:504] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.739913 25546 leader_election.cc:288] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 election: Requested vote from peers f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:41.740435 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:41.740715 24986 raft_consensus.cc:3037] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.740888 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:41.740986 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:41.741194 24596 raft_consensus.cc:3037] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.741273 25247 raft_consensus.cc:3037] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.743263 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a1c9e9695f4e0cbab9d05138eef65a" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:41.743508 24857 raft_consensus.cc:3037] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.832104 25638 raft_consensus.cc:482] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:41.832336 25638 raft_consensus.cc:504] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.834551 25638 leader_election.cc:288] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.837800 24986 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:41.841392 25247 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:41.841887 25053 leader_election.cc:302] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd; no voters:
I20240501 13:59:41.845832 24857 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:41.846191 25546 raft_consensus.cc:2781] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:41.846374 25546 raft_consensus.cc:686] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Becoming Leader. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Running, Role: LEADER
I20240501 13:59:41.846740 25546 consensus_queue.cc:237] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 13:59:41.847478 24596 raft_consensus.cc:2454] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491e21cc623d4879b09975ede207ab2d in term 1.
I20240501 13:59:41.868734 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:41.869079 24727 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:41.870750 25627 raft_consensus.cc:482] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:41.870949 25627 raft_consensus.cc:504] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.871510 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:41.871189 24452 catalog_manager.cc:5484] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 491e21cc623d4879b09975ede207ab2d (127.22.5.197). New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:41.871737 24857 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:41.872282 24923 leader_election.cc:302] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:41.873324 25627 leader_election.cc:288] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.874429 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:41.874670 24596 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:41.875221 25638 raft_consensus.cc:2781] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:41.875362 25638 raft_consensus.cc:482] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:41.875452 25638 raft_consensus.cc:3037] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.880744 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:41.881088 25117 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:41.891981 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:41.892274 25247 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:41.893682 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:41.893946 24986 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:41.894109 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:41.894363 25117 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:41.894363 24790 leader_election.cc:302] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:41.894793 25627 raft_consensus.cc:2781] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:41.894908 25627 raft_consensus.cc:482] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:41.895006 25627 raft_consensus.cc:3037] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.895525 25635 raft_consensus.cc:482] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:41.895727 25635 raft_consensus.cc:504] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.898131 25635 leader_election.cc:288] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.899132 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:41.899348 24596 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:41.907069 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:41.907192 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:41.907400 24986 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:41.908484 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:41.908682 25117 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:41.909425 24532 leader_election.cc:302] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23; no voters:
I20240501 13:59:41.909904 25635 raft_consensus.cc:2781] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:41.910038 25635 raft_consensus.cc:482] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:41.910144 25635 raft_consensus.cc:3037] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.912359 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:41.912559 25247 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:41.939930 25627 raft_consensus.cc:504] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.939952 25638 raft_consensus.cc:504] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.939996 25635 raft_consensus.cc:504] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:41.940479 24857 raft_consensus.cc:2379] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 283529d70ba14547b9440a16350b629e in current term 1: Already voted for candidate f74cb76209d04397a8a206f86e91ac5b in this term.
I20240501 13:59:41.941078 25627 leader_election.cc:288] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Requested vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.941144 25638 leader_election.cc:288] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.942030 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:41.942078 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:41.942281 24727 raft_consensus.cc:3037] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.942394 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:41.942566 24596 raft_consensus.cc:3037] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.944550 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:41.944780 25117 raft_consensus.cc:3037] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.945295 24596 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:41.945533 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfc84e5be47406f95b6246252e8d0f5" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:41.945753 24857 raft_consensus.cc:3037] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.946377 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:41.946588 24986 raft_consensus.cc:3037] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.946615 24595 raft_consensus.cc:2379] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f74cb76209d04397a8a206f86e91ac5b in current term 1: Already voted for candidate 283529d70ba14547b9440a16350b629e in this term.
I20240501 13:59:41.947988 25116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:41.948192 25116 raft_consensus.cc:3037] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.948717 25635 leader_election.cc:288] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Requested vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:41.948767 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:41.949002 25247 raft_consensus.cc:3037] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:41.949103 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:41.949374 24856 raft_consensus.cc:2379] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 283529d70ba14547b9440a16350b629e in current term 1: Already voted for candidate f74cb76209d04397a8a206f86e91ac5b in this term.
I20240501 13:59:41.949658 25115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:41.949676 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:41.950502 24529 leader_election.cc:302] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 283529d70ba14547b9440a16350b629e; no voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b
I20240501 13:59:41.950618 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9762584f4ce94ab68a52bc1f446e569c" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:41.950892 25635 raft_consensus.cc:2726] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:59:42.044560 25117 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:42.044595 25247 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:42.045339 24857 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:42.046258 24923 leader_election.cc:302] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:42.047369 24727 raft_consensus.cc:2454] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:42.047294 24986 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:42.047848 25116 raft_consensus.cc:2454] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:42.047425 25638 raft_consensus.cc:2781] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:42.048365 24790 leader_election.cc:302] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd, f74cb76209d04397a8a206f86e91ac5b; no voters: 283529d70ba14547b9440a16350b629e
I20240501 13:59:42.048696 25627 raft_consensus.cc:2781] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:42.048748 25638 raft_consensus.cc:686] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Becoming Leader. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Running, Role: LEADER
I20240501 13:59:42.049098 25627 raft_consensus.cc:686] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Becoming Leader. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Running, Role: LEADER
I20240501 13:59:42.049350 25638 consensus_queue.cc:237] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:42.049621 25627 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:42.053597 24452 catalog_manager.cc:5484] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:42.053759 24448 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: term changed from 0 to 1, leader changed from <none> to f74cb76209d04397a8a206f86e91ac5b (127.22.5.195). New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:42.226662 25546 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.248570 25546 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.260607 25606 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.261425 25606 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:42.425673 25669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:42.426038 25669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:42.434094 25669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:42.461205 25627 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 13:59:42.478806 25627 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.485193 25627 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.491483 25665 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.540491 25638 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.580102 25638 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.644057 25117 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:42.650682 24727 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:42.657044 25638 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:42.674117 25638 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:42.702755 24922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 51313.4us
I20240501 13:59:42.735980 25726 mvcc.cc:205] Tried to move back new op lower bound from 7022886840703336448 to 7022886838479695872. Current Snapshot: MvccSnapshot[applied={T|T < 7022886840703336448}]
I20240501 13:59:42.795533 25740 mvcc.cc:205] Tried to move back new op lower bound from 7022886840703336448 to 7022886838479695872. Current Snapshot: MvccSnapshot[applied={T|T < 7022886840703336448}]
I20240501 13:59:42.817399 25679 mvcc.cc:205] Tried to move back new op lower bound from 7022886840703336448 to 7022886838479695872. Current Snapshot: MvccSnapshot[applied={T|T < 7022886840703336448 or (T in {7022886840703336448})}]
INSERT report
rows total: 2000
time total: 375.349 ms
time per row: 0.187675 ms
I20240501 13:59:42.974630 24455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:42.985307 24837 tablet_service.cc:1467] Processing CreateTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.985483 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.985618 25227 tablet_service.cc:1467] Processing CreateTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.985805 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.985956 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.986027 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.990034 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.990514 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.996940 25096 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed1065df4344515b0f598f912e22149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.996979 24705 tablet_service.cc:1467] Processing CreateTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:42.997496 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.997531 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.000905 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed1065df4344515b0f598f912e22149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.001562 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.015362 25759 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:43.015204 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed1065df4344515b0f598f912e22149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.015400 25760 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:43.015872 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.016708 25758 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:43.017527 25760 tablet_bootstrap.cc:654] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.018891 25757 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:43.019052 25759 tablet_bootstrap.cc:654] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.020897 25757 tablet_bootstrap.cc:654] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.022042 25758 tablet_bootstrap.cc:654] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.023806 25227 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed1065df4344515b0f598f912e22149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.024268 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed1065df4344515b0f598f912e22149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.024385 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.024881 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.024933 25757 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:43.025097 25757 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 13:59:43.026145 25759 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:43.026034 25757 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.026304 25759 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 13:59:43.026307 25757 raft_consensus.cc:374] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.026396 25757 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.026381 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.026818 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.027325 25759 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.027397 25757 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.027604 25759 raft_consensus.cc:374] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.027693 25759 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.029680 25757 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:43.031630 25758 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:43.032368 25758 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 13:59:43.033609 25758 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.034732 25759 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.035869 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.036358 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.036794 25760 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:43.036962 25760 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240501 13:59:43.037855 25760 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.038098 25760 raft_consensus.cc:374] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.038178 25760 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.038444 25760 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.041222 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.041852 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.041898 25760 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:43.044432 25766 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:43.045766 25758 raft_consensus.cc:374] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.046504 25759 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20240501 13:59:43.046785 25758 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.047618 25766 tablet_bootstrap.cc:654] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.047528 25758 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.051016 25705 raft_consensus.cc:482] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:43.051239 25705 raft_consensus.cc:504] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.052767 25705 leader_election.cc:288] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 13:59:43.052881 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:43.053145 24596 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:43.053192 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:43.053434 25247 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:43.053987 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:43.054256 24987 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:43.054548 25757 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:43.055326 24790 leader_election.cc:302] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:43.056536 25757 tablet_bootstrap.cc:654] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.056556 25760 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:43.058300 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.058300 25227 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:43.058749 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.058750 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:43.059815 25760 tablet_bootstrap.cc:654] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.061100 25705 raft_consensus.cc:2781] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:43.061282 25705 raft_consensus.cc:482] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:43.061385 25705 raft_consensus.cc:3037] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.067900 25759 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:43.070362 25759 tablet_bootstrap.cc:654] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.071622 25760 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:43.071923 25760 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 13:59:43.072975 25760 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.073444 25760 raft_consensus.cc:374] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.073575 25760 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.073985 25760 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.074819 25760 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:43.075330 25761 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:43.077961 25761 tablet_bootstrap.cc:654] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.083818 25758 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.051s user 0.002s sys 0.002s
I20240501 13:59:43.084936 25705 raft_consensus.cc:504] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.086300 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:43.086524 24596 raft_consensus.cc:3037] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.089150 25758 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:43.090736 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:43.090956 25247 raft_consensus.cc:3037] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.091542 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:43.091739 24987 raft_consensus.cc:3037] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.094936 25705 leader_election.cc:288] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 13:59:43.096675 25758 tablet_bootstrap.cc:654] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.096841 25757 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:43.097221 25757 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.018s
I20240501 13:59:43.098379 25757 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.098769 25757 raft_consensus.cc:374] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.098881 25757 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.099304 25757 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.100283 25757 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:43.100353 25766 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:43.100589 25766 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.005s
I20240501 13:59:43.101492 25766 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.101768 25766 raft_consensus.cc:374] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.101871 25766 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.102128 25766 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.102859 25766 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:43.103144 25766 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:43.107594 25766 tablet_bootstrap.cc:654] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.111121 25759 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:43.111328 25759 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.005s
I20240501 13:59:43.111295 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:43.111295 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
W20240501 13:59:43.112154 24792 leader_election.cc:341] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:43.112216 25759 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.112483 25759 raft_consensus.cc:374] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.112568 25759 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
W20240501 13:59:43.112721 24792 leader_election.cc:341] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:43.112901 25759 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.113185 25766 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:43.113709 25759 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:43.114064 25759 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:43.114058 25766 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 13:59:43.114863 25766 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.115149 25766 raft_consensus.cc:374] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.115233 25766 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.115500 25766 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.116830 25766 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:43.117223 25766 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:43.120287 25759 tablet_bootstrap.cc:654] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.120424 25766 tablet_bootstrap.cc:654] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.122308 25758 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:43.122327 25761 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:43.122483 25761 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.012s
I20240501 13:59:43.122509 25758 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.004s
I20240501 13:59:43.123332 25761 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.123324 25758 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.123587 25761 raft_consensus.cc:374] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.123682 25761 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.123694 25758 raft_consensus.cc:374] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.123777 25758 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.123996 25761 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.124059 25758 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.124680 25761 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:43.126150 25758 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 13:59:43.126507 25758 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:43.127070 25766 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:43.127290 25766 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240501 13:59:43.128855 25758 tablet_bootstrap.cc:654] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.129184 25766 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.129499 25766 raft_consensus.cc:374] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.129612 25766 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.129923 25766 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.131304 25759 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:43.131937 25766 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:43.131517 25759 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240501 13:59:43.134145 25759 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.135223 25759 raft_consensus.cc:374] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.135360 25759 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.135632 25759 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.135998 25758 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:43.136155 25758 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 13:59:43.136240 25759 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:43.136680 25761 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:43.136934 25758 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.137228 25758 raft_consensus.cc:374] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.137327 25758 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.137576 25758 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.138140 25758 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:43.139104 25761 tablet_bootstrap.cc:654] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:43.141444 25761 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:43.141630 25761 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:43.142452 25761 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.142712 25761 raft_consensus.cc:374] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:43.142782 25761 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:43.142995 25761 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.143121 25480 raft_consensus.cc:482] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:43.143469 25761 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:43.143345 25480 raft_consensus.cc:504] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.144321 25480 leader_election.cc:288] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 13:59:43.144580 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:43.144799 25117 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:43.144817 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:43.145066 24857 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:43.145068 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:43.145112 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:43.145315 25245 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:43.145356 24595 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:43.145975 24663 leader_election.cc:302] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:43.146597 25480 raft_consensus.cc:2781] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:43.146693 25480 raft_consensus.cc:482] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:43.146800 25480 raft_consensus.cc:3037] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.169358 25768 raft_consensus.cc:482] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:43.169602 25768 raft_consensus.cc:504] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.171798 25768 leader_election.cc:288] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 13:59:43.181605 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:43.181952 24726 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:43.186247 24987 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:43.186713 25247 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:43.187146 24596 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:43.187947 24790 leader_election.cc:302] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 84c5917a503d4a82bb2d9c0d5fdb04bd, f74cb76209d04397a8a206f86e91ac5b; no voters: 909bf3bda47b4c7ab369f460f5e96973
I20240501 13:59:43.188303 25705 raft_consensus.cc:2781] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:43.188429 25705 raft_consensus.cc:686] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Becoming Leader. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Running, Role: LEADER
I20240501 13:59:43.188432 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:43.188696 24986 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:43.188755 25705 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.189488 25180 leader_election.cc:302] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:43.190026 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:43.190119 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:43.190258 25768 raft_consensus.cc:2781] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:43.190268 25117 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:43.190325 24596 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:43.190384 25768 raft_consensus.cc:482] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:43.190477 25768 raft_consensus.cc:3037] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.190763 25480 raft_consensus.cc:504] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.191140 25781 raft_consensus.cc:482] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:43.191283 25781 raft_consensus.cc:504] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:43.191735 25480 leader_election.cc:288] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Requested vote from peers 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 13:59:43.192224 25781 leader_election.cc:288] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:43.192564 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:43.192611 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:43.192710 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:43.192804 24857 raft_consensus.cc:3037] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.192881 25117 raft_consensus.cc:3037] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.194172 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:43.194766 24595 raft_consensus.cc:2379] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 909bf3bda47b4c7ab369f460f5e96973 in current term 1: Already voted for candidate f74cb76209d04397a8a206f86e91ac5b in this term.
I20240501 13:59:43.195931 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:43.196413 25247 raft_consensus.cc:3037] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.196561 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:43.196614 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:43.197194 25245 raft_consensus.cc:2379] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 909bf3bda47b4c7ab369f460f5e96973 in current term 1: Already voted for candidate f74cb76209d04397a8a206f86e91ac5b in this term.
I20240501 13:59:43.197309 24986 raft_consensus.cc:2379] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 909bf3bda47b4c7ab369f460f5e96973 in current term 1: Already voted for candidate f74cb76209d04397a8a206f86e91ac5b in this term.
I20240501 13:59:43.199311 24663 leader_election.cc:302] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 909bf3bda47b4c7ab369f460f5e96973; no voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd
I20240501 13:59:43.199775 25781 raft_consensus.cc:3037] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.200659 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:43.200891 24596 raft_consensus.cc:3037] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.203485 24455 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: term changed from 0 to 1, leader changed from <none> to f74cb76209d04397a8a206f86e91ac5b (127.22.5.195). New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.292461 24857 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:43.292488 25117 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:43.292481 24596 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:43.292526 25781 raft_consensus.cc:2726] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:43.293291 24660 leader_election.cc:302] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 4 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:43.293835 25768 raft_consensus.cc:504] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.294997 25768 leader_election.cc:288] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 election: Requested vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 13:59:43.296017 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:43.296269 24726 raft_consensus.cc:3037] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.297473 25247 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:43.298260 25781 raft_consensus.cc:2781] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:43.298458 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:43.298557 25781 raft_consensus.cc:686] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Becoming Leader. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Running, Role: LEADER
I20240501 13:59:43.298725 25117 raft_consensus.cc:3037] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.299026 25781 consensus_queue.cc:237] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.299546 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:43.299794 24596 raft_consensus.cc:3037] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.303057 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dbc87150c2745c3854334dad7b7ed75" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:43.303297 24986 raft_consensus.cc:3037] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:43.304546 24451 catalog_manager.cc:5484] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.394654 24596 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:43.394672 25117 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:43.394654 24726 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:43.395418 25183 leader_election.cc:302] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:43.395455 24986 raft_consensus.cc:2454] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:43.395843 25768 raft_consensus.cc:2781] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:43.396272 25768 raft_consensus.cc:686] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Becoming Leader. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Running, Role: LEADER
I20240501 13:59:43.396752 25768 consensus_queue.cc:237] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } }
I20240501 13:59:43.400750 24451 catalog_manager.cc:5484] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198). New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.605039 25705 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.633222 25705 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.639621 25705 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.652074 25627 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.704528 25781 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.760851 25781 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.766660 25480 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.767194 25480 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.781407 25790 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.801867 25768 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.808497 25790 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:43.809818 25790 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:43.845490 25792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:43.845964 25792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:43.858433 25792 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
W20240501 13:59:44.099769 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
INSERT report
rows total: 2000
time total: 422.969 ms
time per row: 0.211485 ms
I20240501 13:59:44.437608 24455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:44.446954 25227 tablet_service.cc:1467] Processing CreateTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.447467 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.447602 24836 tablet_service.cc:1467] Processing CreateTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.447685 25096 tablet_service.cc:1467] Processing CreateTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.447692 24967 tablet_service.cc:1467] Processing CreateTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.448060 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.448127 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.448127 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.454159 25095 tablet_service.cc:1467] Processing CreateTablet for tablet dfe9cb33f20b45aa90b066e210804355 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.454432 24706 tablet_service.cc:1467] Processing CreateTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.454604 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.454906 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.454975 24966 tablet_service.cc:1467] Processing CreateTablet for tablet dfe9cb33f20b45aa90b066e210804355 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.455381 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.457531 24705 tablet_service.cc:1467] Processing CreateTablet for tablet dfe9cb33f20b45aa90b066e210804355 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.457723 25226 tablet_service.cc:1467] Processing CreateTablet for tablet dfe9cb33f20b45aa90b066e210804355 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.457957 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.458117 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.462610 24837 tablet_service.cc:1467] Processing CreateTablet for tablet dfe9cb33f20b45aa90b066e210804355 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.462760 24576 tablet_service.cc:1467] Processing CreateTablet for tablet d02fe94840e24205aa98b3190d089784 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.463124 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.463191 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.463816 24707 tablet_service.cc:1467] Processing CreateTablet for tablet d02fe94840e24205aa98b3190d089784 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.464206 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.466465 24965 tablet_service.cc:1467] Processing CreateTablet for tablet d02fe94840e24205aa98b3190d089784 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.466958 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.473140 24835 tablet_service.cc:1467] Processing CreateTablet for tablet d02fe94840e24205aa98b3190d089784 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.473685 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.481815 25096 tablet_service.cc:1467] Processing CreateTablet for tablet d02fe94840e24205aa98b3190d089784 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:44.482491 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.484114 25889 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:44.485244 25892 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:44.487439 25889 tablet_bootstrap.cc:654] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.487669 25892 tablet_bootstrap.cc:654] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.490003 25893 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:44.491111 25890 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:44.491174 25891 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:44.491676 25893 tablet_bootstrap.cc:654] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.492774 25890 tablet_bootstrap.cc:654] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.494246 25888 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:44.494391 25892 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:44.494555 25892 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 13:59:44.495375 25892 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.495703 25891 tablet_bootstrap.cc:654] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.495699 25892 raft_consensus.cc:374] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.495795 25892 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.496093 25892 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.496734 25890 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:44.496881 25890 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 13:59:44.497757 25892 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:44.497675 25890 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.497912 25890 raft_consensus.cc:374] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.498015 25890 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.498067 25892 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:44.498286 25890 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.498977 25890 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:44.499414 25890 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:44.501492 25893 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:44.501435 25888 tablet_bootstrap.cc:654] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.501652 25893 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 13:59:44.502604 25890 tablet_bootstrap.cc:654] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.502990 25892 tablet_bootstrap.cc:654] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.505611 25893 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.505972 25893 raft_consensus.cc:374] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.506081 25893 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.506412 25893 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.507818 25889 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:44.508000 25889 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.000s
I20240501 13:59:44.509020 25889 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.509356 25889 raft_consensus.cc:374] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.509467 25889 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.509770 25889 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.510488 25889 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:44.510838 25889 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:44.510898 25891 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:44.511073 25891 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20240501 13:59:44.512219 25891 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.512570 25891 raft_consensus.cc:374] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.512754 25891 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.512830 25889 tablet_bootstrap.cc:654] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.513068 25891 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.514441 25891 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:44.514904 25891 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:44.516747 25891 tablet_bootstrap.cc:654] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.519824 25890 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:44.520077 25890 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240501 13:59:44.521162 25890 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.521538 25890 raft_consensus.cc:374] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.521622 25890 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.522006 25890 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.522850 25890 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:44.523190 25890 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:44.524276 25889 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:44.524401 25888 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:44.524569 25888 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240501 13:59:44.524626 25889 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20240501 13:59:44.524814 25890 tablet_bootstrap.cc:654] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.525466 25888 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.525760 25888 raft_consensus.cc:374] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.525879 25888 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.526180 25888 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.526566 25889 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.526929 25889 raft_consensus.cc:374] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.527070 25889 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.527436 25889 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.528678 25891 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:44.528847 25891 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240501 13:59:44.529291 25892 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:44.529505 25892 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.005s
I20240501 13:59:44.530335 25889 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 13:59:44.530385 25892 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.530701 25892 raft_consensus.cc:374] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.530742 25889 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:44.530787 25892 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.531040 25892 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.531677 25892 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:44.532333 25888 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 13:59:44.532683 25888 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:44.532939 25889 tablet_bootstrap.cc:654] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.534065 25893 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20240501 13:59:44.535634 25888 tablet_bootstrap.cc:654] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.536808 25890 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:44.537034 25890 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20240501 13:59:44.538066 25890 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.538446 25890 raft_consensus.cc:374] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.538566 25890 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.538928 25890 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.541188 25890 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 13:59:44.543092 25888 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:44.543325 25888 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240501 13:59:44.544190 25888 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.544561 25888 raft_consensus.cc:374] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.544674 25888 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.544479 25891 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.544831 25891 raft_consensus.cc:374] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.544932 25891 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.545768 25891 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.546111 25888 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.546579 25891 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 13:59:44.546998 25891 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:44.547897 25888 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 13:59:44.548228 25888 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:44.549233 25891 tablet_bootstrap.cc:654] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.550841 25888 tablet_bootstrap.cc:654] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:44.553268 25889 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:44.553530 25889 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.009s
I20240501 13:59:44.554600 25889 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.554957 25889 raft_consensus.cc:374] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.555078 25889 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.555583 25888 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:44.555438 25889 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.555727 25888 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 13:59:44.556252 25889 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:44.557008 25888 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.557286 25888 raft_consensus.cc:374] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.557389 25888 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.557634 25888 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.558192 25888 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:44.558571 25891 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:44.558748 25891 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240501 13:59:44.559473 25891 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.559758 25891 raft_consensus.cc:374] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:44.559870 25891 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:44.560148 25891 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.560704 25891 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:44.601871 25768 raft_consensus.cc:482] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:44.602054 25768 raft_consensus.cc:504] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.603021 25768 leader_election.cc:288] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:44.603391 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:44.603391 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:44.603653 25117 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:44.603653 24726 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:44.604110 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:44.604360 24986 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:44.604421 25183 leader_election.cc:302] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:44.604748 25768 raft_consensus.cc:2781] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:44.604887 25768 raft_consensus.cc:482] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:44.604987 25768 raft_consensus.cc:3037] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.607463 25768 raft_consensus.cc:504] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.608783 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:44.608876 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:44.609007 24726 raft_consensus.cc:3037] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.609063 25117 raft_consensus.cc:3037] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.609375 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:44.609571 24986 raft_consensus.cc:3037] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.611073 25768 leader_election.cc:288] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 election: Requested vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:44.611918 25117 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:44.611918 24726 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:44.611900 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:44.612116 24857 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 0.
I20240501 13:59:44.612303 24986 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:44.612341 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe9cb33f20b45aa90b066e210804355" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:44.612578 24856 raft_consensus.cc:3037] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.612705 25180 leader_election.cc:302] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, 84c5917a503d4a82bb2d9c0d5fdb04bd; no voters:
I20240501 13:59:44.612993 25768 raft_consensus.cc:2781] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:44.613157 25768 raft_consensus.cc:686] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Becoming Leader. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Running, Role: LEADER
I20240501 13:59:44.613504 25768 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:44.614954 24856 raft_consensus.cc:2454] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 13:59:44.616720 24447 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198). New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:44.625720 25638 raft_consensus.cc:482] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:44.625945 25638 raft_consensus.cc:504] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.626856 25638 leader_election.cc:288] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:44.627388 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:44.627642 24726 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:44.628129 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:44.628367 24856 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:44.628296 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:44.628597 25117 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:44.628801 24923 leader_election.cc:302] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:44.629184 25638 raft_consensus.cc:2781] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:44.629271 25638 raft_consensus.cc:482] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:44.629374 25638 raft_consensus.cc:3037] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.632122 25638 raft_consensus.cc:504] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.633113 25638 leader_election.cc:288] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 election: Requested vote from peers 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:44.633581 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:44.633903 24726 raft_consensus.cc:3037] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.634135 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:44.634349 25117 raft_consensus.cc:3037] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.636973 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:44.637171 25117 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:44.637219 24856 raft_consensus.cc:3037] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.638700 24726 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:44.639048 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:44.639320 25246 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 0.
I20240501 13:59:44.639262 24922 leader_election.cc:302] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:44.639966 24856 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:44.639911 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c3a66fc5634404a4ee669201d3cc85" candidate_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:44.640166 25246 raft_consensus.cc:3037] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.640352 25638 raft_consensus.cc:2781] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:44.641199 25638 raft_consensus.cc:686] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Becoming Leader. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Running, Role: LEADER
I20240501 13:59:44.641764 25638 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.644084 25246 raft_consensus.cc:2454] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e03e0d09b974c7d8cf7f49652d4fd23 in term 1.
I20240501 13:59:44.650022 24447 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:44.750164 25627 raft_consensus.cc:482] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:44.750440 25627 raft_consensus.cc:504] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.751915 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:44.751897 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:44.751905 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:44.752147 25117 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:44.752147 24726 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:44.752158 24986 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:44.752683 24793 leader_election.cc:302] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:44.752902 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:44.752686 25627 leader_election.cc:288] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:44.753109 24596 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 0.
I20240501 13:59:44.753242 25627 raft_consensus.cc:2781] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:44.753338 25627 raft_consensus.cc:482] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:44.753466 25627 raft_consensus.cc:3037] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.755674 25627 raft_consensus.cc:504] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.756907 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:44.756960 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:44.757040 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:44.757203 24596 raft_consensus.cc:3037] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.757218 24986 raft_consensus.cc:3037] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.757251 24726 raft_consensus.cc:3037] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.760077 24986 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:44.760113 24596 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:44.760161 24726 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:44.760648 24790 leader_election.cc:302] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:44.761013 25918 raft_consensus.cc:2781] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:44.761921 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02fe94840e24205aa98b3190d089784" candidate_uuid: "f74cb76209d04397a8a206f86e91ac5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:44.762166 25117 raft_consensus.cc:3037] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:44.762581 25918 raft_consensus.cc:686] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Becoming Leader. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Running, Role: LEADER
I20240501 13:59:44.762943 25918 consensus_queue.cc:237] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:44.763242 25627 leader_election.cc:288] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:44.764670 25117 raft_consensus.cc:2454] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74cb76209d04397a8a206f86e91ac5b in term 1.
I20240501 13:59:44.767372 24447 catalog_manager.cc:5484] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: term changed from 0 to 1, leader changed from <none> to f74cb76209d04397a8a206f86e91ac5b (127.22.5.195). New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:45.037050 25638 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.038863 25768 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.065168 25768 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.074613 25882 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.084148 25768 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.096666 25880 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.128600 25638 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.160043 25638 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.206715 25918 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.214426 25918 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.225673 25918 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:45.229687 25918 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:45.253298 25921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:45.253787 25921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:45.264047 25921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 372.005 ms
time per row: 0.186003 ms
I20240501 13:59:45.818903 24455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44110:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:45.830107 25226 tablet_service.cc:1467] Processing CreateTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.830593 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.830536 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.830916 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.831110 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.831333 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.837210 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.837859 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.842504 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.843147 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.845387 26019 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:45.846760 24575 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b73ec3b4146fda1248c7cb800afbd (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.847357 26019 tablet_bootstrap.cc:654] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.847366 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b73ec3b4146fda1248c7cb800afbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.848946 26020 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:45.851295 26020 tablet_bootstrap.cc:654] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.853873 26020 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:45.854183 26020 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 13:59:45.854537 24706 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b73ec3b4146fda1248c7cb800afbd (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.855111 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b73ec3b4146fda1248c7cb800afbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.857901 26020 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.858449 26020 raft_consensus.cc:374] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.858635 26020 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.859057 26020 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.860033 26020 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 13:59:45.863058 26023 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:45.866683 26023 tablet_bootstrap.cc:654] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.867544 24835 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b73ec3b4146fda1248c7cb800afbd (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.868021 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b73ec3b4146fda1248c7cb800afbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.867949 25226 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b73ec3b4146fda1248c7cb800afbd (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.867939 24965 tablet_service.cc:1467] Processing CreateTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.868366 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.868359 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b73ec3b4146fda1248c7cb800afbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.873303 24575 tablet_service.cc:1467] Processing CreateTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.873586 25096 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b73ec3b4146fda1248c7cb800afbd (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.873770 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.873726 25095 tablet_service.cc:1467] Processing CreateTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.874042 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b73ec3b4146fda1248c7cb800afbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.874073 25227 tablet_service.cc:1467] Processing CreateTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.874442 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.877370 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.877602 26020 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:45.878731 24705 tablet_service.cc:1467] Processing CreateTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:45.879133 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.879359 26020 tablet_bootstrap.cc:654] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.891095 26028 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:45.894134 26028 tablet_bootstrap.cc:654] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.894949 26026 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:45.895218 26019 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:45.895452 26019 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.012s
I20240501 13:59:45.896454 26023 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:45.896651 26019 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.897015 26019 raft_consensus.cc:374] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.897257 26019 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.897698 26019 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.898082 26024 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:45.898761 26020 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:45.899025 26020 ts_tablet_manager.cc:1367] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.002s
I20240501 13:59:45.899154 26023 ts_tablet_manager.cc:1367] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240501 13:59:45.900053 26024 tablet_bootstrap.cc:654] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.899863 26020 raft_consensus.cc:348] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.900220 26020 raft_consensus.cc:374] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.900310 26020 raft_consensus.cc:729] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.900563 26020 consensus_queue.cc:260] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.901329 26020 ts_tablet_manager.cc:1398] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:45.901664 26020 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
I20240501 13:59:45.903698 26019 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20240501 13:59:45.903802 26020 tablet_bootstrap.cc:654] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.904181 26019 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:45.907362 26019 tablet_bootstrap.cc:654] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.911415 26023 raft_consensus.cc:348] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.913380 26023 raft_consensus.cc:374] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.913533 26023 raft_consensus.cc:729] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.913859 26026 tablet_bootstrap.cc:654] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.914618 26023 consensus_queue.cc:260] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.915441 26024 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:45.915737 26024 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20240501 13:59:45.916836 26019 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:45.916718 26024 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.917039 26024 raft_consensus.cc:374] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.917094 26019 ts_tablet_manager.cc:1367] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20240501 13:59:45.917212 26024 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.917555 26024 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.917966 26019 raft_consensus.cc:348] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.918292 26019 raft_consensus.cc:374] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.918411 26019 raft_consensus.cc:729] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.918732 26019 consensus_queue.cc:260] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.921496 26024 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 13:59:45.922186 26019 ts_tablet_manager.cc:1398] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:45.924165 26024 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: Bootstrap starting.
I20240501 13:59:45.925387 26019 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Bootstrap starting.
I20240501 13:59:45.926079 26020 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: No bootstrap required, opened a new log
I20240501 13:59:45.926280 26020 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.018s
I20240501 13:59:45.926455 26028 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:45.926620 26028 ts_tablet_manager.cc:1367] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240501 13:59:45.927278 26020 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.927445 26028 raft_consensus.cc:348] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.927636 26020 raft_consensus.cc:374] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.927733 26020 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.927778 26028 raft_consensus.cc:374] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.927970 26028 raft_consensus.cc:729] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.928293 26028 consensus_queue.cc:260] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.929306 26019 tablet_bootstrap.cc:654] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.929971 26024 tablet_bootstrap.cc:654] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.931174 26020 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.932039 26020 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240501 13:59:45.933751 26028 ts_tablet_manager.cc:1398] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 13:59:45.935289 26028 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d: Bootstrap starting.
I20240501 13:59:45.937717 26028 tablet_bootstrap.cc:654] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.940212 26023 ts_tablet_manager.cc:1398] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.041s user 0.002s sys 0.007s
I20240501 13:59:45.940657 26023 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:45.940654 26024 tablet_bootstrap.cc:492] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: No bootstrap required, opened a new log
I20240501 13:59:45.940846 26024 ts_tablet_manager.cc:1367] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240501 13:59:45.941733 26024 raft_consensus.cc:348] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.942015 26024 raft_consensus.cc:374] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.942111 26024 raft_consensus.cc:729] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74cb76209d04397a8a206f86e91ac5b, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.942384 26024 consensus_queue.cc:260] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.942809 26023 tablet_bootstrap.cc:654] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.943720 26024 ts_tablet_manager.cc:1398] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:45.946031 26026 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:45.946260 26026 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.009s
I20240501 13:59:45.947162 26026 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.947419 26026 raft_consensus.cc:374] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.947607 26026 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.948040 26026 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.948742 26026 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:45.949156 26026 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23: Bootstrap starting.
I20240501 13:59:45.948431 26019 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: No bootstrap required, opened a new log
I20240501 13:59:45.949581 26019 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.005s
I20240501 13:59:45.950789 26026 tablet_bootstrap.cc:654] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.950598 26019 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.950964 26019 raft_consensus.cc:374] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.952095 26019 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.952829 26026 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23: No bootstrap required, opened a new log
I20240501 13:59:45.952740 26019 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.953146 26026 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 13:59:45.954025 26026 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.954221 26026 raft_consensus.cc:374] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.954303 26026 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e03e0d09b974c7d8cf7f49652d4fd23, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.954538 26026 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.954653 26028 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d: No bootstrap required, opened a new log
I20240501 13:59:45.954962 26028 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240501 13:59:45.955157 26026 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 13:59:45.956562 26028 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.956962 26028 raft_consensus.cc:374] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.957063 26028 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e21cc623d4879b09975ede207ab2d, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.957422 26028 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.959890 26019 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 13:59:45.961176 26028 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 13:59:45.985664 26023 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:45.985888 26023 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.012s
I20240501 13:59:45.986536 26023 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.986752 26023 raft_consensus.cc:374] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.986831 26023 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.987056 26023 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:45.987565 26023 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:45.987912 26023 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 13:59:45.989773 26023 tablet_bootstrap.cc:654] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:45.991780 26023 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: No bootstrap required, opened a new log
I20240501 13:59:45.991912 26023 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:45.992501 26023 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.992697 26023 raft_consensus.cc:374] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:45.992805 26023 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: FOLLOWER
I20240501 13:59:45.993067 26023 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:45.993587 26023 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:45.998363 26032 raft_consensus.cc:482] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:45.998528 26032 raft_consensus.cc:504] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:45.999676 26032 leader_election.cc:288] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:46.000137 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:46.000309 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:46.000420 24856 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.000542 25117 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.000443 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:46.000726 25246 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.001020 24532 leader_election.cc:302] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:46.001374 26032 raft_consensus.cc:2781] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:46.001498 26032 raft_consensus.cc:482] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:46.001607 26032 raft_consensus.cc:3037] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.003952 26032 raft_consensus.cc:504] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:46.004781 26032 leader_election.cc:288] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Requested vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 13:59:46.005236 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:46.005337 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:46.005549 25117 raft_consensus.cc:3037] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.005539 24856 raft_consensus.cc:3037] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.008446 24856 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.008453 25117 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.008926 24532 leader_election.cc:302] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:46.008868 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:46.009152 25246 raft_consensus.cc:3037] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.009337 26032 raft_consensus.cc:2781] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:46.009666 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:46.009910 24726 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.010273 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b73ec3b4146fda1248c7cb800afbd" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:46.010438 24727 raft_consensus.cc:3037] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.012022 25246 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.013007 24727 raft_consensus.cc:2454] T bb4b73ec3b4146fda1248c7cb800afbd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.014204 26032 raft_consensus.cc:686] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Becoming Leader. State: Replica: 283529d70ba14547b9440a16350b629e, State: Running, Role: LEADER
I20240501 13:59:46.014833 26032 consensus_queue.cc:237] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 13:59:46.019061 24447 catalog_manager.cc:5484] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e reported cstate change: term changed from 0 to 1, leader changed from <none> to 283529d70ba14547b9440a16350b629e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:46.077531 25922 raft_consensus.cc:482] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:46.077766 25922 raft_consensus.cc:504] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:46.078711 25922 leader_election.cc:288] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:46.079167 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e" is_pre_election: true
I20240501 13:59:46.079169 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:46.079432 24596 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:46.079430 24987 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:46.079376 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:46.079376 25115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 13:59:46.079631 25115 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:46.079680 25246 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 0.
I20240501 13:59:46.079998 24660 leader_election.cc:302] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:46.080296 25922 raft_consensus.cc:2781] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:46.080395 25922 raft_consensus.cc:482] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:46.080480 25922 raft_consensus.cc:3037] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.082768 25922 raft_consensus.cc:504] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:46.083741 25922 leader_election.cc:288] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Requested vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 13:59:46.083997 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283529d70ba14547b9440a16350b629e"
I20240501 13:59:46.084059 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:46.084300 24987 raft_consensus.cc:3037] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.084303 24596 raft_consensus.cc:3037] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.084721 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:46.084949 25246 raft_consensus.cc:3037] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.085104 25115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "909bf3bda47b4c7ab369f460f5e96973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d"
I20240501 13:59:46.085381 25115 raft_consensus.cc:3037] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.087208 24987 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:46.087208 24596 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:46.087888 24660 leader_election.cc:302] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:46.088083 25246 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:46.088290 25922 raft_consensus.cc:2781] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:46.088507 25115 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bf3bda47b4c7ab369f460f5e96973 in term 1.
I20240501 13:59:46.088848 25922 raft_consensus.cc:686] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Becoming Leader. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Running, Role: LEADER
I20240501 13:59:46.089251 25922 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:46.092614 24447 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.145629 26032 raft_consensus.cc:482] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:46.145889 26032 raft_consensus.cc:504] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:46.146661 26032 leader_election.cc:288] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:46.147161 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973" is_pre_election: true
I20240501 13:59:46.147181 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b" is_pre_election: true
I20240501 13:59:46.147305 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 13:59:46.147315 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 13:59:46.147401 24856 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.147423 24727 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.147552 25246 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.147537 24987 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 0.
I20240501 13:59:46.147903 24532 leader_election.cc:302] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b; no voters:
I20240501 13:59:46.148258 26032 raft_consensus.cc:2781] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:46.148356 26032 raft_consensus.cc:482] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:46.148468 26032 raft_consensus.cc:3037] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.150462 26032 raft_consensus.cc:504] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:46.151400 26032 leader_election.cc:288] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Requested vote from peers 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 13:59:46.151679 25246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd"
I20240501 13:59:46.151733 24727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bf3bda47b4c7ab369f460f5e96973"
I20240501 13:59:46.151921 25246 raft_consensus.cc:3037] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.151880 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23"
I20240501 13:59:46.151952 24727 raft_consensus.cc:3037] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.152084 24987 raft_consensus.cc:3037] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.152606 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "283529d70ba14547b9440a16350b629e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74cb76209d04397a8a206f86e91ac5b"
I20240501 13:59:46.152825 24856 raft_consensus.cc:3037] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:46.154960 24727 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.154973 25246 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.155000 24987 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.155532 24532 leader_election.cc:302] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 283529d70ba14547b9440a16350b629e, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973; no voters:
I20240501 13:59:46.155926 26032 raft_consensus.cc:2781] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:46.156136 26032 raft_consensus.cc:686] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Becoming Leader. State: Replica: 283529d70ba14547b9440a16350b629e, State: Running, Role: LEADER
I20240501 13:59:46.156189 24856 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283529d70ba14547b9440a16350b629e in term 1.
I20240501 13:59:46.156468 26032 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } }
I20240501 13:59:46.159658 24447 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e reported cstate change: term changed from 0 to 1, leader changed from <none> to 283529d70ba14547b9440a16350b629e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:46.408958 26032 consensus_queue.cc:1035] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.505136 25922 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.554337 26032 consensus_queue.cc:1035] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.569628 25922 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.572440 26056 consensus_queue.cc:1035] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.574824 26055 consensus_queue.cc:1035] T bb4b73ec3b4146fda1248c7cb800afbd P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.595242 25922 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.599818 25922 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:46.628257 26048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:46.628631 26048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:46.637359 26048 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240501 13:59:46.650012 24663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 13:59:46.651224 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:35589: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
Using table 'rebalance_test_table_4'
I20240501 13:59:46.701045 26055 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.718312 26032 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.725505 26032 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:46.780053 26032 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:46.786618 26089 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.22.5.254:40379, user_credentials={real_user=slave}} blocked reactor thread for 55532.9us
W20240501 13:59:46.811266 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 84480.6us
INSERT report
rows total: 2000
time total: 461.628 ms
time per row: 0.230814 ms
I20240501 13:59:47.264513 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.199:35589
--local_ip_for_outbound_sockets=127.22.5.199
--tserver_master_addrs=127.22.5.254:40379
--webserver_port=39691
--webserver_interface=127.22.5.199
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:47.629292 26131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:47.629606 26131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:47.629709 26131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:47.638532 26131 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:47.638713 26131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:47.638846 26131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.199
I20240501 13:59:47.648944 26131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.199:35589
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.5.199
--webserver_port=39691
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.199
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:47.650452 26131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:47.652263 26131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:47.667062 26136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:47.668690 26137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:47.675407 26139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:47.676198 26131 server_base.cc:1034] running on GCE node
I20240501 13:59:47.676899 26131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:47.677989 26131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:47.679214 26131 hybrid_clock.cc:648] HybridClock initialized: now 1714571987679152 us; error 54 us; skew 500 ppm
I20240501 13:59:47.679580 26131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:47.682081 26131 webserver.cc:460] Webserver started at http://127.22.5.199:39691/ using document root <none> and password file <none>
I20240501 13:59:47.683064 26131 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:47.683203 26131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:47.688935 26131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:47.691977 26145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:47.693436 26131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:47.693598 26131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2f6d2177839a452eba5e2fc39cb40881"
format_stamp: "Formatted at 2024-05-01 13:59:35 on dist-test-slave-5jzj"
I20240501 13:59:47.694464 26131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:47.712556 26131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:47.713522 26131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:47.713826 26131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:47.714807 26131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:47.716802 26152 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 13:59:47.722285 26131 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 13:59:47.722401 26131 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 13:59:47.722546 26131 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 13:59:47.725673 26131 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 13:59:47.725795 26131 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:47.726132 26152 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 13:59:47.744947 26152 log.cc:824] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:47.749619 26152 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:47.750273 26152 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 13:59:47.751708 26152 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.011s
I20240501 13:59:47.762264 26152 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:47.763211 26152 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: FOLLOWER
I20240501 13:59:47.764030 26152 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } }
I20240501 13:59:47.766106 26152 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20240501 13:59:47.794193 26131 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.199:35589
I20240501 13:59:47.794251 26262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.199:35589 every 8 connection(s)
I20240501 13:59:47.795590 26131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240501 13:59:47.796835 26264 tablet.cc:2369] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:47.800801 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 26131
I20240501 13:59:47.801229 22551 test_util.cc:268] Using random seed: -431598476
I20240501 13:59:47.809497 26263 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 13:59:47.809831 26263 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:47.810801 26263 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 13:59:47.813756 24446 ts_manager.cc:194] Re-registered known tserver with Master: 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 13:59:47.815376 24446 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.199:37795
I20240501 13:59:47.823876 22551 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 13:59:47.824384 22551 test_util.cc:268] Using random seed: -431575326
I20240501 13:59:47.844326 22551 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 13:59:47.845495 22551 test_util.cc:268] Using random seed: -431554221
W20240501 13:59:48.049578 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 49854.6us
W20240501 13:59:48.053261 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 51580.8us
I20240501 13:59:48.097398 22551 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 13:59:48.183499 22551 test_util.cc:268] Using random seed: -431216214
I20240501 13:59:48.823796 26263 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
W20240501 13:59:48.855351 24855 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.006s sys 0.006s
W20240501 13:59:48.855626 24855 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.007s sys 0.007s
W20240501 13:59:48.882515 22551 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.598s user 0.189s sys 0.198s
W20240501 13:59:48.882897 22551 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.598s user 0.189s sys 0.198s
W20240501 13:59:48.896045 24595 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.044s sys 0.096s
W20240501 13:59:48.896190 24595 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.527s user 0.044s sys 0.096s
W20240501 13:59:49.048951 26372 log.cc:924] Time spent T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20240501 13:59:49.222455 22551 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 13:59:49.223068 22551 test_util.cc:268] Using random seed: -430176647
W20240501 13:59:49.344287 26367 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.22.5.254:40379, user_credentials={real_user=slave}} blocked reactor thread for 122174us
I20240501 13:59:49.572971 22551 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 13:59:49.748358 26415 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.22.5.254:40379, user_credentials={real_user=slave}} blocked reactor thread for 65667.3us
W20240501 13:59:50.555995 24518 kernel_stack_watchdog.cc:198] Thread 26446 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7efd3f394980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7efd3c002d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7efd3c0237ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7efd3c01f3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7efd3c01d84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7efd4774dd65 kudu::consensus::Peer::SignalRequest() at ??:0
@ 0x7efd477e147d kudu::consensus::PeerManager::SignalRequest() at ??:0
@ 0x7efd478241d4 kudu::consensus::RaftConsensus::Replicate() at ??:0
@ 0x7efd48b8cb03 kudu::tablet::OpDriver::Prepare() at ??:0
@ 0x7efd48b8b62a kudu::tablet::OpDriver::PrepareTask() at ??:0
@ 0x7efd48b9353c kudu::tablet::OpDriver::ExecuteAsync()::$_1::operator()() at ??:0
@ 0x7efd48b933b9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240501 13:59:50.723184 26446 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.149s sys 0.202s
W20240501 13:59:50.723313 26446 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.149s sys 0.202s
W20240501 13:59:50.733248 26385 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.150s sys 0.242s
W20240501 13:59:50.733389 26385 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.582s user 0.150s sys 0.242s
W20240501 13:59:51.294363 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 45293.5us
W20240501 13:59:52.523733 25050 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 115403us
W20240501 13:59:52.530953 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 124711us
W20240501 13:59:53.000555 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 164679us
W20240501 13:59:53.175334 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 46267us
W20240501 13:59:53.500777 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 49326.2us
W20240501 13:59:53.605307 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 55587.6us
W20240501 13:59:53.793314 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 232010us
W20240501 13:59:54.868489 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 71232.1us
W20240501 13:59:54.922852 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 46660.1us
W20240501 13:59:55.081202 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 80777.7us
I20240501 13:59:55.639235 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.802673 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:55.803051 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:55.892179 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:55.929248 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:56.241729 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:56.295086 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 13:59:56.413704 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 46758.7us
W20240501 13:59:56.767264 25182 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 46587.4us
W20240501 13:59:56.931198 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 44464.7us
W20240501 13:59:56.980527 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 44216.1us
W20240501 13:59:57.149299 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 48006.6us
W20240501 13:59:57.824692 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 72252.6us
W20240501 13:59:58.695652 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 79566us
W20240501 13:59:58.714803 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 48046.5us
W20240501 13:59:58.756071 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 356633us
W20240501 13:59:58.787144 25243 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.000s sys 0.010s
W20240501 13:59:58.787387 25243 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s user 0.000s sys 0.011s
W20240501 13:59:58.904973 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 45499us
I20240501 13:59:59.278609 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:59.286788 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.410749 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.436092 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.438416 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 13:59:59.506167 24662 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.22.5.254:40379, user_credentials={real_user=slave}} blocked reactor thread for 81576.9us
I20240501 13:59:59.516435 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 13:59:59.540603 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 345688us
W20240501 13:59:59.753290 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 72191.1us
W20240501 13:59:59.753595 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 80737.7us
W20240501 13:59:59.836187 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 73246.3us
W20240501 13:59:59.918269 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 54571.1us
W20240501 13:59:59.959561 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 47017.2us
I20240501 13:59:59.995013 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 13:59:59.995692 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 124813us
W20240501 14:00:00.289498 26764 log.cc:924] Time spent T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20240501 14:00:00.429991 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 144939us
W20240501 14:00:00.849385 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 62661.8us
W20240501 14:00:01.374681 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 68919.5us
I20240501 14:00:01.670725 24854 consensus_queue.cc:237] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:01.685935 24726 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:00:01.686617 26470 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:00:01.690266 24591 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:00:01.691905 26470 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:00:01.695889 26781 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:01.696681 24591 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:01.701926 24725 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:01.704037 24447 catalog_manager.cc:5484] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:01.731309 25112 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:00:01.734031 26781 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:00:01.736539 25114 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:01.739238 24983 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:00:01.747385 26781 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
W20240501 14:00:01.821534 24793 consensus_peers.cc:485] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d02fe94840e24205aa98b3190d089784. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:01.858105 25243 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } }
I20240501 14:00:01.861703 24983 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 14:00:01.862289 26720 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
W20240501 14:00:01.867748 25180 consensus_peers.cc:485] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfe9cb33f20b45aa90b066e210804355. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:01.869711 24854 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 14:00:01.871803 25112 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 14:00:01.875487 24982 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:01.885720 25050 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 47103.1us
I20240501 14:00:01.915707 26777 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 14:00:01.917405 26777 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 14:00:01.931281 24725 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 14:00:01.971861 26775 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
I20240501 14:00:01.982463 24854 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
I20240501 14:00:01.994196 24451 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
I20240501 14:00:02.003057 26720 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 14:00:02.005332 24981 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
I20240501 14:00:02.020952 25112 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
I20240501 14:00:02.037371 24725 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } } }
W20240501 14:00:02.342895 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 53785us
I20240501 14:00:02.355024 26821 ts_tablet_manager.cc:902] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
W20240501 14:00:02.535617 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 62030.9us
I20240501 14:00:02.538280 24724 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
W20240501 14:00:02.542403 24663 consensus_peers.cc:485] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6b2ff2919a045aabb3c320be4b4fcfd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:02.543390 26821 tablet_copy_client.cc:308] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:37091
I20240501 14:00:02.543876 24853 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240501 14:00:02.545209 26649 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240501 14:00:02.557593 24594 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240501 14:00:02.562983 25242 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240501 14:00:02.566303 26749 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240501 14:00:02.561820 26824 ts_tablet_manager.cc:902] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Initiating tablet copy from peer 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 14:00:02.602186 26649 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240501 14:00:02.617223 24986 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240501 14:00:02.629014 26749 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240501 14:00:02.633607 26653 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:02.636564 24590 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:02.643644 24446 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:02.680167 25244 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:02.685966 24986 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:02.761720 24854 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:02.979832 24877 tablet_copy_service.cc:140] P f74cb76209d04397a8a206f86e91ac5b: Received BeginTabletCopySession request for tablet d02fe94840e24205aa98b3190d089784 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54459)
I20240501 14:00:02.980307 24877 tablet_copy_service.cc:161] P f74cb76209d04397a8a206f86e91ac5b: Beginning new tablet copy session on tablet d02fe94840e24205aa98b3190d089784 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54459: session id = 2f6d2177839a452eba5e2fc39cb40881-d02fe94840e24205aa98b3190d089784
I20240501 14:00:03.014480 24877 tablet_copy_source_session.cc:205] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:03.102659 26821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02fe94840e24205aa98b3190d089784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.124969 26821 tablet_copy_client.cc:783] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:03.128479 26821 tablet_copy_client.cc:647] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:03.133590 26851 ts_tablet_manager.cc:902] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 14:00:03.238155 26851 tablet_copy_client.cc:308] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:00:03.256618 26821 tablet_copy_client.cc:523] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:03.259542 24986 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:03.260587 26821 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:03.329252 24726 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:03.329934 26758 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 14:00:03.354162 25117 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
W20240501 14:00:03.384912 26824 thread.cc:636] tablet-download-pool-dfe9cb33f20b45aa90b066e210804355 [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.039s sys 0.099s
W20240501 14:00:03.385211 26824 thread.cc:602] tablet-download-pool-dfe9cb33f20b45aa90b066e210804355 [worker] (thread pool) Time spent starting thread: real 0.608s user 0.039s sys 0.099s
I20240501 14:00:03.418887 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet c6b2ff2919a045aabb3c320be4b4fcfd from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54681)
I20240501 14:00:03.419878 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet c6b2ff2919a045aabb3c320be4b4fcfd from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54681: session id = 2f6d2177839a452eba5e2fc39cb40881-c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:00:03.441511 24747 tablet_copy_source_session.cc:205] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:03.444247 26851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.458631 26851 tablet_copy_client.cc:783] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:03.458952 26851 tablet_copy_client.cc:647] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:03.562922 26824 tablet_copy_client.cc:308] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:32877
I20240501 14:00:03.564169 25267 tablet_copy_service.cc:140] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Received BeginTabletCopySession request for tablet dfe9cb33f20b45aa90b066e210804355 from peer 283529d70ba14547b9440a16350b629e ({username='slave'} at 127.22.5.193:55259)
I20240501 14:00:03.564445 25267 tablet_copy_service.cc:161] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Beginning new tablet copy session on tablet dfe9cb33f20b45aa90b066e210804355 from peer 283529d70ba14547b9440a16350b629e at {username='slave'} at 127.22.5.193:55259: session id = 283529d70ba14547b9440a16350b629e-dfe9cb33f20b45aa90b066e210804355
I20240501 14:00:03.569737 25267 tablet_copy_source_session.cc:205] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:03.590893 26824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.599826 25246 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:03.601433 26577 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:03.601804 26755 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 14:00:03.605561 24853 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:03.633113 26851 tablet_copy_client.cc:523] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:03.636833 26851 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:03.640317 26755 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 14:00:03.659937 26755 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:03.668776 24453 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:03.681640 26824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.22.5.198:54327 (request call id 185) took 1196 ms. Trace:
I20240501 14:00:03.701776 25246 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:03.726116 25114 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:03.681802 26824 rpcz_store.cc:276] 0501 14:00:02.485492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:00:02.485542 (+ 50us) service_pool.cc:225] Handling call
0501 14:00:02.569810 (+ 84268us) ts_tablet_manager.cc:903] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Initiating tablet copy from peer 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
0501 14:00:03.574432 (+1004622us) tablet_copy_client.cc:353] Tablet copy session begun
0501 14:00:03.676726 (+102294us) tablet_metadata.cc:641] Metadata flushed
0501 14:00:03.676962 (+ 236us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 14:00:03.681075 (+ 4113us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 14:00:03.681385 (+ 310us) tablet_replica.cc:205] Creating consensus instance
0501 14:00:03.681514 (+ 129us) ts_tablet_manager.cc:958] Replica registered.
0501 14:00:03.681596 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":104,"tablet-copy.queue_time_us":75638,"thread_start_us":1071836,"threads_started":5}
I20240501 14:00:03.754418 26824 tablet_copy_client.cc:783] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:03.754820 26824 tablet_copy_client.cc:647] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:03.776053 26824 tablet_copy_client.cc:523] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:03.779247 24852 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:03.791479 26824 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Bootstrap starting.
I20240501 14:00:03.752120 24723 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:03.949740 24923 consensus_peers.cc:485] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7c3a66fc5634404a4ee669201d3cc85. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:04.058312 24590 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
W20240501 14:00:04.092576 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 181019us
I20240501 14:00:04.127141 24853 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 14:00:04.127915 26873 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240501 14:00:04.128712 24981 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 14:00:04.136302 26455 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240501 14:00:04.145545 26455 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.150213 24986 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.223186 24723 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 14:00:04.233352 26873 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240501 14:00:04.236037 24722 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.242316 24453 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:04.248103 24922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 111628us
W20240501 14:00:04.282088 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 155820us
I20240501 14:00:04.429940 24853 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.453953 25246 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
W20240501 14:00:04.456557 26824 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.049s sys 0.055s
W20240501 14:00:04.467592 26824 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.635s user 0.049s sys 0.055s
I20240501 14:00:04.484333 26872 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240501 14:00:04.489674 25244 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.564275 26910 ts_tablet_manager.cc:902] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 14:00:04.652312 26910 tablet_copy_client.cc:308] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:43227
I20240501 14:00:04.662427 24854 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:04.666603 25241 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240501 14:00:04.667026 24596 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240501 14:00:04.667351 26781 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240501 14:00:04.668458 26753 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240501 14:00:04.671092 26657 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.688943 24596 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.693689 24447 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:04.694118 25241 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:04.720813 24722 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240501 14:00:04.727241 26657 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240501 14:00:04.732762 24724 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:04.848260 25180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 70938.4us
I20240501 14:00:04.878533 26821 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:04.881584 26821 tablet_bootstrap.cc:492] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:04.882027 26821 ts_tablet_manager.cc:1367] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 1.622s user 0.163s sys 0.006s
I20240501 14:00:04.883438 26821 raft_consensus.cc:348] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:04.884503 26821 raft_consensus.cc:729] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:04.884868 26821 consensus_queue.cc:260] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:04.892365 26821 ts_tablet_manager.cc:1398] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240501 14:00:04.894603 24877 tablet_copy_service.cc:342] P f74cb76209d04397a8a206f86e91ac5b: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-d02fe94840e24205aa98b3190d089784 received from {username='slave'} at 127.22.5.199:54459
I20240501 14:00:04.894783 24877 tablet_copy_service.cc:433] P f74cb76209d04397a8a206f86e91ac5b: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-d02fe94840e24205aa98b3190d089784 on tablet d02fe94840e24205aa98b3190d089784 with peer 2f6d2177839a452eba5e2fc39cb40881
W20240501 14:00:04.999377 25183 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 222914us
I20240501 14:00:05.019536 26824 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=720 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:05.023583 26824 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Bootstrap complete.
I20240501 14:00:05.024669 26824 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Time spent bootstrapping tablet: real 1.233s user 0.213s sys 0.066s
I20240501 14:00:05.026589 26824 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } }
I20240501 14:00:05.027606 26824 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 283529d70ba14547b9440a16350b629e, State: Initialized, Role: LEARNER
I20240501 14:00:05.028100 26824 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: true } }
I20240501 14:00:05.029384 26824 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:05.040669 25267 tablet_copy_service.cc:342] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Request end of tablet copy session 283529d70ba14547b9440a16350b629e-dfe9cb33f20b45aa90b066e210804355 received from {username='slave'} at 127.22.5.193:55259
I20240501 14:00:05.040870 25267 tablet_copy_service.cc:433] P 84c5917a503d4a82bb2d9c0d5fdb04bd: ending tablet copy session 283529d70ba14547b9440a16350b629e-dfe9cb33f20b45aa90b066e210804355 on tablet dfe9cb33f20b45aa90b066e210804355 with peer 283529d70ba14547b9440a16350b629e
I20240501 14:00:05.066514 26218 raft_consensus.cc:1203] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.79->[1.80-1.90] Dedup: 1.86->[1.87-1.90]
I20240501 14:00:05.100116 24986 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240501 14:00:05.328714 24594 raft_consensus.cc:1203] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.92] Dedup: 1.89->[1.90-1.92]
I20240501 14:00:05.419510 26781 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
W20240501 14:00:05.530560 24793 consensus_peers.cc:485] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f7c6d19fc7402ebc9fa3a3f07f76e0. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:00:05.546176 24532 consensus_peers.cc:485] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94749eeb6ba44d3ea0803cacf47017ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:05.546480 25007 tablet_copy_service.cc:140] P 4e03e0d09b974c7d8cf7f49652d4fd23: Received BeginTabletCopySession request for tablet f7c3a66fc5634404a4ee669201d3cc85 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:39731)
I20240501 14:00:05.546753 25007 tablet_copy_service.cc:161] P 4e03e0d09b974c7d8cf7f49652d4fd23: Beginning new tablet copy session on tablet f7c3a66fc5634404a4ee669201d3cc85 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:39731: session id = 2f6d2177839a452eba5e2fc39cb40881-f7c3a66fc5634404a4ee669201d3cc85
W20240501 14:00:05.555208 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 68779us
W20240501 14:00:05.574721 24663 consensus_peers.cc:485] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 14:00:05.579573 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 76877us
I20240501 14:00:05.598090 24980 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:05.626593 25007 tablet_copy_source_session.cc:205] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:05.629509 26910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:05.645519 26951 ts_tablet_manager.cc:902] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 14:00:05.653755 26910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.22.5.196:58745 (request call id 1) took 1102 ms. Trace:
I20240501 14:00:05.654632 26910 rpcz_store.cc:276] 0501 14:00:04.551633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:00:04.551684 (+ 51us) service_pool.cc:225] Handling call
0501 14:00:04.564514 (+ 12830us) ts_tablet_manager.cc:903] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
0501 14:00:05.628848 (+1064334us) tablet_copy_client.cc:353] Tablet copy session begun
0501 14:00:05.632534 (+ 3686us) tablet_metadata.cc:641] Metadata flushed
0501 14:00:05.632658 (+ 124us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 14:00:05.653432 (+ 20774us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 14:00:05.653583 (+ 151us) tablet_replica.cc:205] Creating consensus instance
0501 14:00:05.653679 (+ 96us) ts_tablet_manager.cc:958] Replica registered.
0501 14:00:05.653704 (+ 25us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":97,"tablet-copy.queue_time_us":12335,"thread_start_us":99892,"threads_started":5}
I20240501 14:00:05.657068 26910 tablet_copy_client.cc:783] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:05.657577 26910 tablet_copy_client.cc:647] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:05.698109 26910 tablet_copy_client.cc:523] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:00:05.744410 25183 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 62250.5us
I20240501 14:00:05.766875 26910 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:05.870329 26959 raft_consensus.cc:1050] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: attempting to promote NON_VOTER 283529d70ba14547b9440a16350b629e to VOTER
I20240501 14:00:05.897212 26959 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } }
I20240501 14:00:05.949503 24596 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 LEARNER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:05.950253 26961 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:05.954017 26970 ts_tablet_manager.cc:902] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 14:00:05.960748 25117 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:05.960935 24980 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:05.961519 26962 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:05.962731 26954 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:05.968577 24853 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:05.982601 24722 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:05.997352 26959 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.000591 26781 raft_consensus.cc:1050] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:06.002488 26781 consensus_queue.cc:237] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:06.009378 26218 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:00:06.011157 26931 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:00:06.017668 24723 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:00:06.020748 26781 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:00:06.022271 25117 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:00:06.023279 26931 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:00:06.023944 24595 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:00:06.024576 26931 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:00:06.041146 26959 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.042120 26954 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.045420 26753 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.047632 24981 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:00:06.048265 26657 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 14:00:06.048983 24984 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.050669 24447 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:06.051609 24722 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.054857 24454 catalog_manager.cc:5484] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:06.056689 25117 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:06.066401 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 61627.9us
I20240501 14:00:06.062583 24593 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.072079 25114 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.076783 26218 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.090238 24595 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.106349 24986 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 80 to 93, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.123116 24853 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.129379 26951 tablet_copy_client.cc:308] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35547
I20240501 14:00:06.140146 26970 tablet_copy_client.cc:308] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:37091
I20240501 14:00:06.143040 24877 tablet_copy_service.cc:140] P f74cb76209d04397a8a206f86e91ac5b: Received BeginTabletCopySession request for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54459)
I20240501 14:00:06.143237 24877 tablet_copy_service.cc:161] P f74cb76209d04397a8a206f86e91ac5b: Beginning new tablet copy session on tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54459: session id = 2f6d2177839a452eba5e2fc39cb40881-11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:00:06.147832 24877 tablet_copy_source_session.cc:205] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:06.152840 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:06.152922 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 51876.1us
I20240501 14:00:06.154125 24724 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 83 to 94, 283529d70ba14547b9440a16350b629e (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.157150 24853 consensus_queue.cc:237] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:06.157362 25247 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } }
I20240501 14:00:06.159324 26970 tablet_copy_client.cc:783] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:06.163765 25114 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:06.165426 24981 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:06.166421 26961 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:00:06.168499 26961 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:00:06.164108 26217 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:06.169907 26970 tablet_copy_client.cc:647] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:06.169802 26781 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.174379 26961 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.175311 26970 tablet_copy_client.cc:523] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:06.179294 26851 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=135 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:06.181876 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfe9cb33f20b45aa90b066e210804355 with cas_config_opid_index 94: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:06.183522 25114 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.184404 24980 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.190332 26851 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:06.190778 26851 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 2.554s user 0.222s sys 0.024s
I20240501 14:00:06.191354 24453 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:06.198982 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe9cb33f20b45aa90b066e210804355 with delete_type TABLET_DATA_TOMBSTONED (TS f74cb76209d04397a8a206f86e91ac5b not found in new config with opid_index 96) from {username='slave'} at 127.0.0.1:48194
I20240501 14:00:06.191594 26851 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:06.209551 26970 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:06.218209 26851 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:06.219053 26851 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:06.221766 26851 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240501 14:00:06.223755 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-c6b2ff2919a045aabb3c320be4b4fcfd received from {username='slave'} at 127.22.5.199:54681
I20240501 14:00:06.223955 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-c6b2ff2919a045aabb3c320be4b4fcfd on tablet c6b2ff2919a045aabb3c320be4b4fcfd with peer 2f6d2177839a452eba5e2fc39cb40881
I20240501 14:00:06.228979 24983 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:06.230060 25111 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:06.230044 26989 tablet_replica.cc:330] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: stopping tablet replica
I20240501 14:00:06.230715 26989 raft_consensus.cc:2229] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:06.230917 26753 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.232715 26989 raft_consensus.cc:2258] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:06.233331 24589 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:06.234755 24592 raft_consensus.cc:1261] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:00:06.235412 26753 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.238248 26753 consensus_queue.cc:1035] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:00:06.241442 26989 ts_tablet_manager.cc:1875] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:06.243737 26753 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.250242 24591 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.260030 26961 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:00:06.260473 24446 catalog_manager.cc:5484] T d02fe94840e24205aa98b3190d089784 P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.262512 24596 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.264710 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet d02fe94840e24205aa98b3190d089784 with delete_type TABLET_DATA_TOMBSTONED (TS 909bf3bda47b4c7ab369f460f5e96973 not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:51724
I20240501 14:00:06.268406 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d02fe94840e24205aa98b3190d089784 with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:06.270282 26217 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.271767 26989 ts_tablet_manager.cc:1888] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.95
I20240501 14:00:06.271936 26989 log.cc:1196] T dfe9cb33f20b45aa90b066e210804355 P f74cb76209d04397a8a206f86e91ac5b: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/dfe9cb33f20b45aa90b066e210804355
I20240501 14:00:06.276631 24442 catalog_manager.cc:4830] TS f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): tablet dfe9cb33f20b45aa90b066e210804355 (table rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]) successfully deleted
I20240501 14:00:06.326884 24616 tablet_copy_service.cc:140] P 283529d70ba14547b9440a16350b629e: Received BeginTabletCopySession request for tablet 94749eeb6ba44d3ea0803cacf47017ea from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:58237)
I20240501 14:00:06.327195 24616 tablet_copy_service.cc:161] P 283529d70ba14547b9440a16350b629e: Beginning new tablet copy session on tablet 94749eeb6ba44d3ea0803cacf47017ea from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:58237: session id = 2f6d2177839a452eba5e2fc39cb40881-94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:00:06.331596 24616 tablet_copy_source_session.cc:205] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:06.333550 26951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:06.339984 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 114033us
I20240501 14:00:06.347819 26951 tablet_copy_client.cc:783] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:06.348119 26951 tablet_copy_client.cc:647] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:06.356289 26951 tablet_copy_client.cc:523] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:06.360656 24724 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:00:06.361091 26951 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:06.362915 25111 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.417873 26961 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:00:06.448575 26997 tablet_replica.cc:330] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: stopping tablet replica
I20240501 14:00:06.449510 26997 raft_consensus.cc:2229] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:06.451462 26997 pending_rounds.cc:62] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Trying to abort 1 pending ops.
I20240501 14:00:06.451640 26997 pending_rounds.cc:68] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Aborting op as it isn't in flight: id { term: 1 index: 94 } timestamp: 7022886936826490880 op_type: WRITE_OP write_request { tablet_id: "d02fe94840e24205aa98b3190d089784" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\316U\002\017x\334\320D\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""37f802b3f6f2e69a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022886936161005568 authz_token { token_data: "\010\201\230\311\261\006\"3\n\005slave\022*\n e9a223c9990f4a2abf67a7ef7e640f99\020\001\030\001 \001(\001" signature: "<redacted>""[\215\315\260\226\024\314\224\013f\214{\236\027L3\311\244\254=\020\341\370*\214\353\261]3\202\232\330A\303\'\335j\245\0232r#z\216\364f\210\310\t\021\314\2009j2\206gjm\314\265\260\260\331" signing_key_seq_num: 0 } } request_id { client_id: "1d8df74044e24869967045621797e99c" seq_no: 273 first_incomplete_seq_no: 273 attempt_no: 0 }
I20240501 14:00:06.452921 26997 raft_consensus.cc:2258] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:06.491511 24981 raft_consensus.cc:2932] T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:06.529322 26218 raft_consensus.cc:1203] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.129->[1.130-1.165] Dedup: 1.136->[1.137-1.165]
I20240501 14:00:06.523669 24723 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 94 to 96, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } }
I20240501 14:00:06.549934 26997 ts_tablet_manager.cc:1875] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:06.611851 26997 ts_tablet_manager.cc:1888] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.94
I20240501 14:00:06.612160 26997 log.cc:1196] T d02fe94840e24205aa98b3190d089784 P 909bf3bda47b4c7ab369f460f5e96973: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/d02fe94840e24205aa98b3190d089784
I20240501 14:00:06.613742 24441 catalog_manager.cc:4830] TS 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): tablet d02fe94840e24205aa98b3190d089784 (table rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]) successfully deleted
W20240501 14:00:07.111615 26958 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.001s sys 0.009s
W20240501 14:00:07.111857 26958 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.509s user 0.001s sys 0.009s
W20240501 14:00:07.589632 24947 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.002s sys 0.012s
W20240501 14:00:07.590030 24947 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.657s user 0.002s sys 0.012s
I20240501 14:00:07.604539 26891 raft_consensus.cc:1050] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:07.606271 26891 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.174, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:07.610944 24592 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 14:00:07.611750 26866 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240501 14:00:07.657399 24980 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 14:00:07.692482 26217 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 14:00:07.696861 24855 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 14:00:07.699648 25241 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 14:00:07.703629 26891 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240501 14:00:07.743369 26866 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 170, Time since last communication: 0.000s
I20240501 14:00:07.757906 26721 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240501 14:00:07.760789 26721 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 173, Time since last communication: 0.000s
I20240501 14:00:07.832984 26891 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.175: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:07.839529 24453 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:07.865665 24596 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:07.876107 26910 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=784 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:07.887853 24855 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:07.894119 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 101034us
I20240501 14:00:07.897652 24984 raft_consensus.cc:1203] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.171->[1.172-1.175] Dedup: 1.174->[1.175-1.175]
I20240501 14:00:07.898733 26910 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:07.899184 26910 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 2.132s user 0.159s sys 0.023s
I20240501 14:00:07.899982 26910 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:07.880591 26218 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:07.917219 26910 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:07.917562 26910 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:07.941875 26910 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240501 14:00:07.942831 25007 tablet_copy_service.cc:342] P 4e03e0d09b974c7d8cf7f49652d4fd23: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-f7c3a66fc5634404a4ee669201d3cc85 received from {username='slave'} at 127.22.5.199:39731
I20240501 14:00:07.943025 25007 tablet_copy_service.cc:433] P 4e03e0d09b974c7d8cf7f49652d4fd23: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-f7c3a66fc5634404a4ee669201d3cc85 on tablet f7c3a66fc5634404a4ee669201d3cc85 with peer 2f6d2177839a452eba5e2fc39cb40881
W20240501 14:00:07.959990 24923 consensus_peers.cc:485] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:00:07.992190 24723 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 176, Committed index: 176, Last appended: 1.176, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:08.016299 26218 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 14:00:08.017346 26891 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 14:00:08.025102 24854 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 14:00:08.025839 27055 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 14:00:08.028687 27055 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.177: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.034508 24854 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.038259 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6b2ff2919a045aabb3c320be4b4fcfd with cas_config_opid_index 175: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:08.045126 26218 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.047710 24446 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:08.055737 24965 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd with delete_type TABLET_DATA_TOMBSTONED (TS 4e03e0d09b974c7d8cf7f49652d4fd23 not found in new config with opid_index 177) from {username='slave'} at 127.0.0.1:49768
I20240501 14:00:08.082790 27059 tablet_replica.cc:330] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: stopping tablet replica
I20240501 14:00:08.084131 27059 raft_consensus.cc:2229] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:08.098695 27059 pending_rounds.cc:62] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Trying to abort 1 pending ops.
I20240501 14:00:08.098950 27059 pending_rounds.cc:68] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Aborting op as it isn't in flight: id { term: 1 index: 175 } timestamp: 7022886943151812608 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" old_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } } new_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } } }
I20240501 14:00:08.099661 27059 raft_consensus.cc:2894] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Aborting config change with OpId 1.175: Aborted: Op aborted
I20240501 14:00:08.101868 27059 raft_consensus.cc:2866] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:08.102100 27059 raft_consensus.cc:2258] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:08.110296 24588 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 14:00:08.118721 26891 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 14:00:08.122804 24588 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.139581 27059 ts_tablet_manager.cc:1875] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:08.189033 26970 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=117 overwritten=0 applied=117 ignored=0} inserts{seen=802 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:08.196815 26970 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:08.199016 26970 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 2.005s user 0.216s sys 0.022s
I20240501 14:00:08.199919 25247 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 14:00:08.200357 26970 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
W20240501 14:00:08.213429 24793 consensus_peers.cc:485] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:00:08.203639 25244 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 130 to 175, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.239647 26970 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:08.240289 26970 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 117, Last appended: 1.117, Last appended by leader: 117, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:08.244185 26970 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240501 14:00:08.245286 24877 tablet_copy_service.cc:342] P f74cb76209d04397a8a206f86e91ac5b: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-11f7c6d19fc7402ebc9fa3a3f07f76e0 received from {username='slave'} at 127.22.5.199:54459
I20240501 14:00:08.245465 24877 tablet_copy_service.cc:433] P f74cb76209d04397a8a206f86e91ac5b: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-11f7c6d19fc7402ebc9fa3a3f07f76e0 on tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with peer 2f6d2177839a452eba5e2fc39cb40881
W20240501 14:00:08.278292 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 93457.2us
I20240501 14:00:08.299561 27055 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 174, Time since last communication: 0.000s
W20240501 14:00:08.351778 26951 log.cc:924] Time spent T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20240501 14:00:08.374292 25180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 98324.3us
I20240501 14:00:08.536496 27059 ts_tablet_manager.cc:1888] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.175
I20240501 14:00:08.542572 27059 log.cc:1196] T c6b2ff2919a045aabb3c320be4b4fcfd P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:00:08.555893 24439 catalog_manager.cc:4830] TS 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): tablet c6b2ff2919a045aabb3c320be4b4fcfd (table rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]) successfully deleted
W20240501 14:00:08.639699 24532 consensus_peers.cc:485] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:00:08.670959 26218 raft_consensus.cc:1203] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.113->[1.114-1.129] Dedup: 1.117->[1.118-1.129]
I20240501 14:00:08.688014 25246 raft_consensus.cc:1203] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.175->[1.176-1.181] Dedup: 1.176->[1.177-1.181]
I20240501 14:00:08.692778 25246 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 175 to 177, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:08.698873 26951 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=124 overwritten=0 applied=124 ignored=0} inserts{seen=809 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:08.699424 26951 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:08.699751 26951 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 2.339s user 0.193s sys 0.023s
I20240501 14:00:08.700557 26951 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:08.700907 26951 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:08.701179 26951 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 124, Last appended: 1.124, Last appended by leader: 124, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:08.702090 26951 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:08.702826 24616 tablet_copy_service.cc:342] P 283529d70ba14547b9440a16350b629e: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-94749eeb6ba44d3ea0803cacf47017ea received from {username='slave'} at 127.22.5.199:58237
I20240501 14:00:08.702991 24616 tablet_copy_service.cc:433] P 283529d70ba14547b9440a16350b629e: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-94749eeb6ba44d3ea0803cacf47017ea on tablet 94749eeb6ba44d3ea0803cacf47017ea with peer 2f6d2177839a452eba5e2fc39cb40881
I20240501 14:00:08.718104 26217 raft_consensus.cc:1203] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.108] Dedup: 1.99->[1.100-1.108]
W20240501 14:00:08.856815 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 141402us
W20240501 14:00:09.015972 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 279635us
W20240501 14:00:09.106791 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 46563.6us
I20240501 14:00:09.162249 26216 raft_consensus.cc:1203] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.107->[1.108-1.155] Dedup: 1.124->[1.125-1.155]
W20240501 14:00:09.189415 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 49376.2us
I20240501 14:00:09.205757 27031 raft_consensus.cc:1050] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:09.206780 27031 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:09.311116 26931 raft_consensus.cc:1050] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:09.312034 26931 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 1.135, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:09.315536 24722 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240501 14:00:09.316025 25246 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240501 14:00:09.316179 26215 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240501 14:00:09.316588 27022 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 134, Time since last communication: 0.000s
I20240501 14:00:09.316972 27067 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 133, Time since last communication: 0.000s
I20240501 14:00:09.320485 27067 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 134, Time since last communication: 0.000s
I20240501 14:00:09.320650 24591 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240501 14:00:09.321185 27067 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 134, Time since last communication: 0.000s
I20240501 14:00:09.323392 26217 raft_consensus.cc:1203] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.133->[1.134-1.135] Dedup: 1.135->[]
I20240501 14:00:09.326678 27067 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.330679 25241 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.331290 24723 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:00:09.332079 27074 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.000s
I20240501 14:00:09.335038 24446 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.335572 24590 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:09.341785 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 160763us
I20240501 14:00:09.342260 26217 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.351440 25111 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:00:09.368008 26215 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:00:09.368583 25246 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:00:09.365713 24722 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.444032 27060 raft_consensus.cc:1050] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:09.445037 27060 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 1.164, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:09.451786 24854 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240501 14:00:09.457417 26872 consensus_queue.cc:1037] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 1.164, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240501 14:00:09.469802 24722 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240501 14:00:09.471354 24980 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240501 14:00:09.472107 27000 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20240501 14:00:09.472834 25242 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240501 14:00:09.474695 27000 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240501 14:00:09.483389 26872 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.483960 24722 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.489415 24855 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.499056 25241 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.515476 24982 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240501 14:00:09.516952 27000 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 14:00:09.524137 26216 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240501 14:00:09.524935 27000 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240501 14:00:09.527500 26218 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.538929 24981 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.556023 25111 consensus_queue.cc:237] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:09.530761 27091 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.131s
I20240501 14:00:09.568341 24590 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:00:09.569872 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:00:09.571879 27030 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.000s
I20240501 14:00:09.574412 27092 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 109, Time since last communication: 0.001s
I20240501 14:00:09.580471 24984 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:00:09.581511 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:00:09.587057 26771 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:09.589939 24980 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:09.606009 24588 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:09.626567 27065 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 133, Time since last communication: 0.002s
I20240501 14:00:09.645582 25241 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:00:09.646703 24854 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 14:00:09.646988 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:00:09.648960 25241 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:09.653185 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 14:00:09.660864 24854 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:09.712038 26218 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.716935 26788 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.729144 24448 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 108 to 165, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.733544 24447 catalog_manager.cc:5484] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index -1 to 253, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:09.748999 24453 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 reported cstate change: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.764890 24591 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 168, Committed index: 168, Last appended: 1.168, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:09.772691 26218 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 14:00:09.773408 26872 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 168, Time since last communication: 0.000s
I20240501 14:00:09.778364 24855 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 14:00:09.778970 26872 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240501 14:00:09.784685 24982 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 14:00:09.785851 26872 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240501 14:00:09.788996 24724 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:09.789721 25053 consensus_peers.cc:485] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96a1c9e9695f4e0cbab9d05138eef65a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:09.797364 24723 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 14:00:09.798785 27064 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240501 14:00:09.801857 27064 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.169: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.807410 24723 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.821054 24439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94749eeb6ba44d3ea0803cacf47017ea with cas_config_opid_index 165: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:09.823709 24454 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.830721 24854 raft_consensus.cc:1203] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.167->[1.168-1.168] Dedup: 1.168->[]
I20240501 14:00:09.834360 26216 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.837905 25112 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.841517 25226 tablet_service.cc:1514] Processing DeleteTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea with delete_type TABLET_DATA_TOMBSTONED (TS 84c5917a503d4a82bb2d9c0d5fdb04bd not found in new config with opid_index 169) from {username='slave'} at 127.0.0.1:45540
I20240501 14:00:09.858042 24986 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.887233 24853 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 165 to 169, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.905103 24982 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:09.915938 24727 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 14:00:09.920331 27031 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 111, Time since last communication: 0.004s
I20240501 14:00:09.949102 24984 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index 114 to 136, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:09.947958 24724 raft_consensus.cc:1203] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.111->[1.112-1.112] Dedup: 1.112->[]
I20240501 14:00:09.983668 25246 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.046990 24981 raft_consensus.cc:1203] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.167->[1.168-1.168] Dedup: 1.168->[]
I20240501 14:00:10.031741 24853 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 96 to 111, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.106326 26217 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 14:00:10.123162 27117 ts_tablet_manager.cc:902] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 14:00:10.135192 25244 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 14:00:10.172767 26788 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 14:00:10.180068 27117 tablet_copy_client.cc:308] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:35069
I20240501 14:00:10.217733 27111 tablet_replica.cc:330] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: stopping tablet replica
I20240501 14:00:10.218816 27111 raft_consensus.cc:2229] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:10.219326 27111 raft_consensus.cc:2258] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:00:10.221341 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 47513.7us
I20240501 14:00:10.236570 25137 tablet_copy_service.cc:140] P 491e21cc623d4879b09975ede207ab2d: Received BeginTabletCopySession request for tablet 96a1c9e9695f4e0cbab9d05138eef65a from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54893)
I20240501 14:00:10.236843 25137 tablet_copy_service.cc:161] P 491e21cc623d4879b09975ede207ab2d: Beginning new tablet copy session on tablet 96a1c9e9695f4e0cbab9d05138eef65a from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54893: session id = 2f6d2177839a452eba5e2fc39cb40881-96a1c9e9695f4e0cbab9d05138eef65a
I20240501 14:00:10.240742 25137 tablet_copy_source_session.cc:205] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:10.243029 27117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a1c9e9695f4e0cbab9d05138eef65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.259848 27117 tablet_copy_client.cc:783] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:10.260452 27117 tablet_copy_client.cc:647] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:10.263866 27117 tablet_copy_client.cc:523] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:10.278843 27117 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
I20240501 14:00:10.358932 24852 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 14:00:10.379072 24855 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 1.143, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:10.389374 26216 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 143. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240501 14:00:10.390918 26931 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I20240501 14:00:10.415005 24591 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 143. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240501 14:00:10.415671 27065 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I20240501 14:00:10.423594 26931 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.144: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.428886 24594 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:10.435966 25243 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.025s sys 0.042s
I20240501 14:00:10.435599 24454 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:10.436106 25243 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.580s user 0.025s sys 0.042s
I20240501 14:00:10.437592 25241 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 141. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240501 14:00:10.439771 24981 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240501 14:00:10.462009 27092 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 14:00:10.465514 27031 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.470971 24439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7c3a66fc5634404a4ee669201d3cc85 with cas_config_opid_index 111: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:10.477102 24452 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:10.483153 24722 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.487525 27067 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 140, Time since last communication: 0.000s
I20240501 14:00:10.491576 25243 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.496723 27065 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 14:00:10.515820 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with delete_type TABLET_DATA_TOMBSTONED (TS 909bf3bda47b4c7ab369f460f5e96973 not found in new config with opid_index 144) from {username='slave'} at 127.0.0.1:51724
I20240501 14:00:10.516599 27140 tablet_replica.cc:330] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: stopping tablet replica
I20240501 14:00:10.516899 27140 raft_consensus.cc:2229] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:10.517594 27140 raft_consensus.cc:2258] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:10.519695 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 with delete_type TABLET_DATA_TOMBSTONED (TS 491e21cc623d4879b09975ede207ab2d not found in new config with opid_index 113) from {username='slave'} at 127.0.0.1:57230
I20240501 14:00:10.520767 27141 tablet_replica.cc:330] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: stopping tablet replica
I20240501 14:00:10.521466 27141 raft_consensus.cc:2229] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:10.522543 27141 raft_consensus.cc:2258] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:10.529832 27141 ts_tablet_manager.cc:1875] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:10.531172 27140 ts_tablet_manager.cc:1875] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:10.539506 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with cas_config_opid_index 136: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:10.542860 26217 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:10.547541 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 56544.4us
I20240501 14:00:10.575385 27111 ts_tablet_manager.cc:1875] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:00:10.584841 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 119749us
I20240501 14:00:10.585345 27031 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 14:00:10.586100 27141 ts_tablet_manager.cc:1888] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.111
I20240501 14:00:10.586275 27141 log.cc:1196] T f7c3a66fc5634404a4ee669201d3cc85 P 491e21cc623d4879b09975ede207ab2d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:00:10.587277 24442 catalog_manager.cc:4830] TS 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): tablet f7c3a66fc5634404a4ee669201d3cc85 (table rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]) successfully deleted
I20240501 14:00:10.683869 24852 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.698904 25246 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:10.703342 27140 ts_tablet_manager.cc:1888] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.143
I20240501 14:00:10.739636 27140 log.cc:1196] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 909bf3bda47b4c7ab369f460f5e96973: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:00:10.751598 24441 catalog_manager.cc:4830] TS 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (table rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]) successfully deleted
I20240501 14:00:10.779978 27111 ts_tablet_manager.cc:1888] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.168
I20240501 14:00:10.798714 27111 log.cc:1196] T 94749eeb6ba44d3ea0803cacf47017ea P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:00:10.806569 24442 catalog_manager.cc:4830] TS 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877): tablet 94749eeb6ba44d3ea0803cacf47017ea (table rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]) successfully deleted
I20240501 14:00:11.009470 24980 raft_consensus.cc:1203] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.139->[1.140-1.144] Dedup: 1.142->[1.143-1.144]
I20240501 14:00:11.011445 24980 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 136 to 144, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.063577 26217 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:11.230880 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 62049us
I20240501 14:00:11.275362 27117 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=255 overwritten=0 applied=255 ignored=0} inserts{seen=886 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:11.275945 27117 tablet_bootstrap.cc:492] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:11.276310 27117 ts_tablet_manager.cc:1367] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 0.998s user 0.332s sys 0.061s
I20240501 14:00:11.277503 27117 raft_consensus.cc:348] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:11.277839 27117 raft_consensus.cc:729] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:11.278126 27117 consensus_queue.cc:260] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 255, Last appended: 1.255, Last appended by leader: 255, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:11.278806 27117 ts_tablet_manager.cc:1398] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:11.286425 25137 tablet_copy_service.cc:342] P 491e21cc623d4879b09975ede207ab2d: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-96a1c9e9695f4e0cbab9d05138eef65a received from {username='slave'} at 127.22.5.199:54893
I20240501 14:00:11.286650 25137 tablet_copy_service.cc:433] P 491e21cc623d4879b09975ede207ab2d: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-96a1c9e9695f4e0cbab9d05138eef65a on tablet 96a1c9e9695f4e0cbab9d05138eef65a with peer 2f6d2177839a452eba5e2fc39cb40881
W20240501 14:00:11.318504 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 134991us
W20240501 14:00:11.326792 24591 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.006s sys 0.024s
W20240501 14:00:11.326933 24591 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.598s user 0.006s sys 0.024s
W20240501 14:00:11.367532 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 48306.8us
W20240501 14:00:11.436030 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 58712.7us
I20240501 14:00:11.544790 26216 raft_consensus.cc:1203] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.252->[1.253-1.269] Dedup: 1.255->[1.256-1.269]
W20240501 14:00:11.758610 24854 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.999s user 0.000s sys 0.016s
W20240501 14:00:11.760676 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 152048us
W20240501 14:00:11.772068 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 67675.5us
W20240501 14:00:11.897225 24854 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.041s user 0.000s sys 0.016s
W20240501 14:00:11.913862 26457 log.cc:924] Time spent T bb4b73ec3b4146fda1248c7cb800afbd P f74cb76209d04397a8a206f86e91ac5b: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20240501 14:00:11.924898 26771 raft_consensus.cc:1050] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:11.926231 26771 consensus_queue.cc:237] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:11.930521 24983 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240501 14:00:11.930536 25242 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240501 14:00:11.930804 26216 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240501 14:00:11.931129 27115 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:00:11.931684 24595 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240501 14:00:11.931715 26306 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:00:11.932709 27115 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:00:11.933210 26306 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:00:11.938403 26306 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.939137 24984 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.940567 24854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.5.194:42105 (request call id 665) took 1181 ms. Trace:
I20240501 14:00:11.942788 26216 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.948649 25242 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.949988 24452 catalog_manager.cc:5484] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 reported cstate change: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.955947 24588 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:11.962157 24855 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240501 14:00:11.966131 26306 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240501 14:00:11.959344 24854 rpcz_store.cc:276] 0501 14:00:10.716576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:00:10.717190 (+ 614us) service_pool.cc:225] Handling call
0501 14:00:10.717309 (+ 119us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 14:00:10.717379 (+ 70us) raft_consensus.cc:1520] Early marking committed up to index 206
0501 14:00:11.897428 (+1180049us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 14:00:11.897450 (+ 22us) raft_consensus.cc:1631] Marking committed up to 206
0501 14:00:11.897461 (+ 11us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 14:00:11.897483 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 14:00:11.897536 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 14:00:11.992337 24853 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 253 to 270, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.139717 25112 consensus_queue.cc:237] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:12.143874 24594 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240501 14:00:12.144069 24987 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240501 14:00:12.144724 26306 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240501 14:00:12.146032 26306 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240501 14:00:12.148912 27116 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.150599 25246 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240501 14:00:12.151304 26216 raft_consensus.cc:1261] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240501 14:00:12.151836 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240501 14:00:12.153803 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96a1c9e9695f4e0cbab9d05138eef65a with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:12.153936 25246 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.154065 24987 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.158957 24594 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.161002 24451 catalog_manager.cc:5484] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:12.164790 26771 consensus_queue.cc:1035] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240501 14:00:12.168498 26216 raft_consensus.cc:2932] T 96a1c9e9695f4e0cbab9d05138eef65a P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 270 to 272, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:12.190135 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a1c9e9695f4e0cbab9d05138eef65a with delete_type TABLET_DATA_TOMBSTONED (TS f74cb76209d04397a8a206f86e91ac5b not found in new config with opid_index 272) from {username='slave'} at 127.0.0.1:48194
W20240501 14:00:12.194756 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 45232.8us
I20240501 14:00:12.203979 27212 tablet_replica.cc:330] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: stopping tablet replica
I20240501 14:00:12.204293 27212 raft_consensus.cc:2229] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:12.205364 27212 raft_consensus.cc:2258] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:12.213390 27212 ts_tablet_manager.cc:1875] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:12.243572 27212 ts_tablet_manager.cc:1888] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.271
I20240501 14:00:12.243820 27212 log.cc:1196] T 96a1c9e9695f4e0cbab9d05138eef65a P f74cb76209d04397a8a206f86e91ac5b: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/96a1c9e9695f4e0cbab9d05138eef65a
W20240501 14:00:12.308596 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 54143.3us
I20240501 14:00:12.353739 24442 catalog_manager.cc:4830] TS f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): tablet 96a1c9e9695f4e0cbab9d05138eef65a (table rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]) successfully deleted
W20240501 14:00:12.579002 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 56895.1us
W20240501 14:00:12.794793 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 94253.8us
W20240501 14:00:12.795382 25180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 169569us
W20240501 14:00:13.190876 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 65738.8us
W20240501 14:00:13.249783 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 54072.6us
W20240501 14:00:13.434243 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 54936.7us
W20240501 14:00:13.497972 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 213958us
W20240501 14:00:13.792521 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 98469.1us
W20240501 14:00:14.098500 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 135681us
I20240501 14:00:14.193631 24723 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.202, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:14.212807 25112 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
W20240501 14:00:14.216908 24663 consensus_peers.cc:485] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0a8a92803e1459ca3338c8054b2f8b6. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:00:14.217259 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 53254.7us
I20240501 14:00:14.217248 26721 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I20240501 14:00:14.218708 24984 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:14.226615 24590 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:14.227733 27155 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I20240501 14:00:14.229398 26721 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 200, Time since last communication: 0.000s
I20240501 14:00:14.231961 27155 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.203: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.236311 25244 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:14.237143 27197 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I20240501 14:00:14.240003 24454 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:14.253490 24590 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.267755 25111 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.288071 24987 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.294730 25244 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index -1 to 203, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.899472 25111 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
W20240501 14:00:14.905377 25053 consensus_peers.cc:485] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f64ab2203c9042989dbcab79bab164a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:14.908314 24596 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 14:00:14.908947 27173 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240501 14:00:14.911435 24856 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 14:00:14.911720 24986 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 14:00:14.916347 27173 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240501 14:00:14.920863 26771 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240501 14:00:14.922632 26771 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.933250 24854 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:14.933817 24451 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:14.951737 24588 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:15.057482 24982 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:15.058363 24726 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 14:00:15.059316 27173 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240501 14:00:15.061870 24726 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index -1 to 213, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:15.354418 27304 ts_tablet_manager.cc:902] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
W20240501 14:00:15.365409 26215 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.071s sys 0.156s
W20240501 14:00:15.365976 26215 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.513s user 0.071s sys 0.156s
I20240501 14:00:15.572346 27316 ts_tablet_manager.cc:902] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 14:00:15.635816 27316 tablet_copy_client.cc:308] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:35069
I20240501 14:00:15.636595 25137 tablet_copy_service.cc:140] P 491e21cc623d4879b09975ede207ab2d: Received BeginTabletCopySession request for tablet f64ab2203c9042989dbcab79bab164a6 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54893)
I20240501 14:00:15.636794 25137 tablet_copy_service.cc:161] P 491e21cc623d4879b09975ede207ab2d: Beginning new tablet copy session on tablet f64ab2203c9042989dbcab79bab164a6 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54893: session id = 2f6d2177839a452eba5e2fc39cb40881-f64ab2203c9042989dbcab79bab164a6
I20240501 14:00:15.639509 25137 tablet_copy_source_session.cc:205] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:15.640902 27316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.655476 27316 tablet_copy_client.cc:783] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:15.674227 27316 tablet_copy_client.cc:647] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:15.685719 27316 tablet_copy_client.cc:523] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:15.692590 27316 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
W20240501 14:00:15.803808 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 63565.9us
I20240501 14:00:15.991042 27304 tablet_copy_client.cc:308] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:00:15.992480 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet c0a8a92803e1459ca3338c8054b2f8b6 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54681)
I20240501 14:00:15.992687 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet c0a8a92803e1459ca3338c8054b2f8b6 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54681: session id = 2f6d2177839a452eba5e2fc39cb40881-c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:00:15.995759 24747 tablet_copy_source_session.cc:205] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:16.001219 27304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:16.110213 27304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.22.5.194:36471 (request call id 175) took 1257 ms. Trace:
I20240501 14:00:16.116138 27304 rpcz_store.cc:276] 0501 14:00:14.852610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:00:14.853233 (+ 623us) service_pool.cc:225] Handling call
0501 14:00:15.360539 (+507306us) ts_tablet_manager.cc:903] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
0501 14:00:16.000541 (+640002us) tablet_copy_client.cc:353] Tablet copy session begun
0501 14:00:16.101865 (+101324us) tablet_metadata.cc:641] Metadata flushed
0501 14:00:16.101999 (+ 134us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 14:00:16.109884 (+ 7885us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 14:00:16.110033 (+ 149us) tablet_replica.cc:205] Creating consensus instance
0501 14:00:16.110142 (+ 109us) ts_tablet_manager.cc:958] Replica registered.
0501 14:00:16.110170 (+ 28us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":95,"tablet-copy.queue_time_us":500916,"thread_start_us":1143053,"threads_started":5}
I20240501 14:00:16.215543 27304 tablet_copy_client.cc:783] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:16.220122 27304 tablet_copy_client.cc:647] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:16.227741 27304 tablet_copy_client.cc:523] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:16.415838 27304 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
W20240501 14:00:16.427258 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 152543us
I20240501 14:00:16.430351 24852 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:16.435367 25112 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240501 14:00:16.436067 27247 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240501 14:00:16.439855 25242 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
W20240501 14:00:16.440186 24793 consensus_peers.cc:485] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9762584f4ce94ab68a52bc1f446e569c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:16.440800 27247 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240501 14:00:16.444212 27142 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:16.448594 25242 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:16.454305 25112 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:16.457386 24451 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:16.483430 24588 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
W20240501 14:00:16.484628 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 45545.4us
I20240501 14:00:16.559612 24984 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240501 14:00:16.565903 27246 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 229, Time since last communication: 0.000s
W20240501 14:00:16.628342 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 62537.4us
I20240501 14:00:16.694649 27142 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 229, Time since last communication: 0.000s
I20240501 14:00:16.704245 24588 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:16.739251 27343 log.cc:924] Time spent T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20240501 14:00:16.791234 24984 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index -1 to 235, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:16.823315 27359 ts_tablet_manager.cc:902] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
W20240501 14:00:17.050354 27347 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.102s sys 0.204s
W20240501 14:00:17.050509 27347 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.102s sys 0.204s
W20240501 14:00:17.141924 27361 log.cc:924] Time spent T d02fe94840e24205aa98b3190d089784 P 4e03e0d09b974c7d8cf7f49652d4fd23: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
W20240501 14:00:17.177414 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 128108us
W20240501 14:00:17.188886 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 138274us
I20240501 14:00:17.379730 27359 tablet_copy_client.cc:308] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:37091
I20240501 14:00:17.380726 24877 tablet_copy_service.cc:140] P f74cb76209d04397a8a206f86e91ac5b: Received BeginTabletCopySession request for tablet 9762584f4ce94ab68a52bc1f446e569c from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54459)
I20240501 14:00:17.380887 24877 tablet_copy_service.cc:161] P f74cb76209d04397a8a206f86e91ac5b: Beginning new tablet copy session on tablet 9762584f4ce94ab68a52bc1f446e569c from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54459: session id = 2f6d2177839a452eba5e2fc39cb40881-9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:17.383805 24877 tablet_copy_source_session.cc:205] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:17.394030 27359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.416189 27359 tablet_copy_client.cc:783] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:17.416474 27359 tablet_copy_client.cc:647] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:17.419674 27359 tablet_copy_client.cc:523] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:17.442648 27359 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
W20240501 14:00:17.458736 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 76769.3us
W20240501 14:00:17.637507 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 80303.1us
I20240501 14:00:17.745640 24725 consensus_queue.cc:237] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:17.760676 25246 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 1 index: 187. (index mismatch)
I20240501 14:00:17.760720 24595 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 1 index: 187. (index mismatch)
I20240501 14:00:17.761297 26891 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240501 14:00:17.762567 24853 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 1 index: 187. (index mismatch)
W20240501 14:00:17.763406 24663 consensus_peers.cc:485] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ed1065df4344515b0f598f912e22149. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:17.763759 27099 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240501 14:00:17.764292 27099 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240501 14:00:17.770220 27403 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.186: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:17.771416 25246 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.186: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:17.779691 24452 catalog_manager.cc:5484] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:17.780947 24853 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.186: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
I20240501 14:00:17.783200 24595 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.186: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:17.805624 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 65651.2us
W20240501 14:00:17.862334 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 56290us
W20240501 14:00:17.895170 25052 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 314793us
W20240501 14:00:17.901304 25053 consensus_peers.cc:485] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:00:17.903261 25110 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 184. Preceding OpId from leader: term: 1 index: 187. (index mismatch)
I20240501 14:00:17.904560 27403 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 183, Time since last communication: 0.000s
I20240501 14:00:17.915452 25116 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.186: config changed from index -1 to 186, NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } } }
W20240501 14:00:18.001817 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 49439.5us
W20240501 14:00:18.132196 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 61442.7us
W20240501 14:00:18.305246 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 65429.7us
I20240501 14:00:18.435701 27432 ts_tablet_manager.cc:902] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
W20240501 14:00:18.537009 27359 log.cc:924] Time spent T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Append to log took a long time: real 0.060s user 0.004s sys 0.000s
W20240501 14:00:18.619370 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 54234us
I20240501 14:00:18.735978 27432 tablet_copy_client.cc:308] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:00:18.737069 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet 9ed1065df4344515b0f598f912e22149 from peer 2f6d2177839a452eba5e2fc39cb40881 ({username='slave'} at 127.22.5.199:54681)
I20240501 14:00:18.737282 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet 9ed1065df4344515b0f598f912e22149 from peer 2f6d2177839a452eba5e2fc39cb40881 at {username='slave'} at 127.22.5.199:54681: session id = 2f6d2177839a452eba5e2fc39cb40881-9ed1065df4344515b0f598f912e22149
I20240501 14:00:18.740208 24747 tablet_copy_source_session.cc:205] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:18.742062 27432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed1065df4344515b0f598f912e22149. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:18.749280 27304 log.cc:924] Time spent T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Append to log took a long time: real 0.064s user 0.002s sys 0.000s
I20240501 14:00:18.759486 27432 tablet_copy_client.cc:783] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:18.759817 27432 tablet_copy_client.cc:647] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:18.767405 27432 tablet_copy_client.cc:523] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:18.771071 27432 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap starting.
W20240501 14:00:18.792398 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 60465.1us
W20240501 14:00:19.010046 24922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 48389.4us
W20240501 14:00:19.014465 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 54689.3us
I20240501 14:00:19.127154 27432 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=196 overwritten=0 applied=195 ignored=0} inserts{seen=826 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:19.127789 27432 tablet_bootstrap.cc:492] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:19.128194 27432 ts_tablet_manager.cc:1367] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 0.357s user 0.234s sys 0.043s
I20240501 14:00:19.129180 27432 raft_consensus.cc:348] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.130764 27432 raft_consensus.cc:729] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:19.131058 27432 consensus_queue.cc:260] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 195, Last appended: 1.196, Last appended by leader: 196, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.140400 27432 ts_tablet_manager.cc:1398] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20240501 14:00:19.142366 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-9ed1065df4344515b0f598f912e22149 received from {username='slave'} at 127.22.5.199:54681
I20240501 14:00:19.142526 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-9ed1065df4344515b0f598f912e22149 on tablet 9ed1065df4344515b0f598f912e22149 with peer 2f6d2177839a452eba5e2fc39cb40881
I20240501 14:00:19.265769 26216 raft_consensus.cc:1203] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.185->[1.186-1.200] Dedup: 1.196->[1.197-1.200]
W20240501 14:00:19.380328 24663 consensus_peers.cc:485] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:00:19.506359 27359 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=243 overwritten=0 applied=243 ignored=0} inserts{seen=928 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:19.509585 27359 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:19.509970 27359 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 2.067s user 0.320s sys 0.017s
I20240501 14:00:19.513963 27359 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.521209 27359 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:19.521576 27359 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 243, Last appended: 1.243, Last appended by leader: 243, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.522245 27359 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:00:19.524340 24877 tablet_copy_service.cc:342] P f74cb76209d04397a8a206f86e91ac5b: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-9762584f4ce94ab68a52bc1f446e569c received from {username='slave'} at 127.22.5.199:54459
I20240501 14:00:19.524456 24877 tablet_copy_service.cc:433] P f74cb76209d04397a8a206f86e91ac5b: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-9762584f4ce94ab68a52bc1f446e569c on tablet 9762584f4ce94ab68a52bc1f446e569c with peer 2f6d2177839a452eba5e2fc39cb40881
I20240501 14:00:19.594234 26721 raft_consensus.cc:1050] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:19.595738 26721 consensus_queue.cc:237] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:19.600009 24856 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:19.602591 27331 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:00:19.603307 25111 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:19.604435 25246 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:19.605283 26891 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I20240501 14:00:19.605764 26891 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:00:19.608719 26216 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:19.616117 27331 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:19.617910 27402 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:00:19.621294 24590 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:19.622531 26215 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:19.624612 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 444238us
I20240501 14:00:19.629112 25111 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:19.635454 27197 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:00:19.639354 24591 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:19.640256 24451 catalog_manager.cc:5484] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:19.639560 25242 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:19.644876 24852 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 186 to 203, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:19.781555 27304 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=215 overwritten=0 applied=215 ignored=0} inserts{seen=887 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:19.782096 27304 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:19.782495 27304 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 3.367s user 0.269s sys 0.038s
I20240501 14:00:19.785310 27304 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.785637 27304 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:19.830623 27304 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 215, Last appended: 1.215, Last appended by leader: 215, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:19.837685 27304 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20240501 14:00:19.842348 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-c0a8a92803e1459ca3338c8054b2f8b6 received from {username='slave'} at 127.22.5.199:54681
I20240501 14:00:19.842535 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-c0a8a92803e1459ca3338c8054b2f8b6 on tablet c0a8a92803e1459ca3338c8054b2f8b6 with peer 2f6d2177839a452eba5e2fc39cb40881
I20240501 14:00:19.882342 26218 raft_consensus.cc:1203] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.202->[1.203-1.249] Dedup: 1.215->[1.216-1.249]
I20240501 14:00:19.952432 26216 raft_consensus.cc:1203] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.234->[1.235-1.273] Dedup: 1.243->[1.244-1.273]
I20240501 14:00:20.041530 24726 consensus_queue.cc:237] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:20.097597 25242 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240501 14:00:20.097909 25112 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240501 14:00:20.098837 27402 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:00:20.106583 24852 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240501 14:00:20.168169 27331 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:00:20.168895 27331 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:00:20.236382 26213 raft_consensus.cc:1261] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240501 14:00:20.273326 26721 consensus_queue.cc:1035] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 206, Time since last communication: 0.000s
I20240501 14:00:20.274505 27403 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.329283 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ed1065df4344515b0f598f912e22149 with cas_config_opid_index 203: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:00:20.352093 25053 consensus_peers.cc:485] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d -> Peer 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): Couldn't send request to peer 2f6d2177839a452eba5e2fc39cb40881. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20240501 14:00:20.363062 24447 catalog_manager.cc:5484] T 9ed1065df4344515b0f598f912e22149 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:20.376627 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed1065df4344515b0f598f912e22149 with delete_type TABLET_DATA_TOMBSTONED (TS 283529d70ba14547b9440a16350b629e not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:44100
I20240501 14:00:20.429545 24853 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.430773 25247 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.443883 27495 tablet_replica.cc:330] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: stopping tablet replica
I20240501 14:00:20.444429 27495 raft_consensus.cc:2229] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:20.444798 27495 pending_rounds.cc:62] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Trying to abort 1 pending ops.
I20240501 14:00:20.441617 25111 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.445653 27495 pending_rounds.cc:68] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Aborting op as it isn't in flight: id { term: 1 index: 207 } timestamp: 7022886993308938240 op_type: WRITE_OP write_request { tablet_id: "9ed1065df4344515b0f598f912e22149" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000~i\3660R\233\244\201\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""26f50fc8c5be908d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022886993238716416 authz_token { token_data: "\010\200\230\311\261\006\"3\n\005slave\022*\n f42e7adb897e4d9b9877c7894b6c028e\020\001\030\001 \001(\001" signature: "<redacted>""Ol\224\215\317VZe\346\373\325\345\210\317\037\024J\2737\013\227\034g\271|5\213:\277}G\014h9\221\347\214\362\274\365\305U\330\013\331w\257\240!\363p\303J\274O\207\027.\237\210\274\301A\336" signing_key_seq_num: 0 } } request_id { client_id: "06a1cd901c19420da4b6fb619715d9da" seq_no: 547 first_incomplete_seq_no: 547 attempt_no: 0 }
I20240501 14:00:20.450138 27495 raft_consensus.cc:2258] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:20.453989 27495 ts_tablet_manager.cc:1875] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:20.475648 27497 raft_consensus.cc:1050] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:20.481631 26211 raft_consensus.cc:2932] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 203 to 208, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:20.499624 27439 consensus_queue.cc:1159] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "491e21cc623d4879b09975ede207ab2d" responder_term: 1 status { last_received { term: 1 index: 274 } last_committed_idx: 274 last_received_current_leader { term: 1 index: 274 } }
W20240501 14:00:20.502151 27142 consensus_queue.cc:1159] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" responder_term: 1 status { last_received { term: 1 index: 274 } last_committed_idx: 273 last_received_current_leader { term: 1 index: 274 } }
I20240501 14:00:20.506407 27497 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:20.511809 26211 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240501 14:00:20.512467 25242 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240501 14:00:20.512508 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20240501 14:00:20.513535 27495 ts_tablet_manager.cc:1888] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.207
I20240501 14:00:20.513696 27495 log.cc:1196] T 9ed1065df4344515b0f598f912e22149 P 283529d70ba14547b9440a16350b629e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ed1065df4344515b0f598f912e22149
I20240501 14:00:20.514605 24439 catalog_manager.cc:4830] TS 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): tablet 9ed1065df4344515b0f598f912e22149 (table rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]) successfully deleted
I20240501 14:00:20.515924 24987 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240501 14:00:20.518843 27491 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 273, Time since last communication: 0.000s
I20240501 14:00:20.520344 24588 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240501 14:00:20.521034 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20240501 14:00:20.527185 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20240501 14:00:20.528136 25112 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240501 14:00:20.530408 27497 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
W20240501 14:00:20.532569 25183 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 75608us
I20240501 14:00:20.541961 27497 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.547523 25112 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.566195 26211 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.569294 27403 raft_consensus.cc:1050] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:20.568641 25246 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.570916 27403 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.255, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:20.576725 24985 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240501 14:00:20.578435 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:00:20.579888 25247 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240501 14:00:20.580734 25117 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240501 14:00:20.585350 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:00:20.580122 24452 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:20.604156 26214 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240501 14:00:20.631982 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:00:20.649216 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 253, Time since last communication: 0.000s
I20240501 14:00:20.674249 24594 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240501 14:00:20.683184 27403 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:00:20.688364 24588 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.692974 24856 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.278, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:20.694696 24987 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 235 to 275, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.698482 25111 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240501 14:00:20.699076 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 276, Time since last communication: 0.000s
I20240501 14:00:20.701779 25246 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240501 14:00:20.705427 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 277, Time since last communication: 0.000s
I20240501 14:00:20.705989 26216 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240501 14:00:20.707126 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 277, Time since last communication: 0.000s
I20240501 14:00:20.710765 27498 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.279: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.713559 25246 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.719158 24452 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:20.720700 26216 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.721381 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9762584f4ce94ab68a52bc1f446e569c with cas_config_opid_index 275: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:20.731976 24590 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20240501 14:00:20.736196 24965 tablet_service.cc:1514] Processing DeleteTablet for tablet 9762584f4ce94ab68a52bc1f446e569c with delete_type TABLET_DATA_TOMBSTONED (TS 4e03e0d09b974c7d8cf7f49652d4fd23 not found in new config with opid_index 279) from {username='slave'} at 127.0.0.1:49768
I20240501 14:00:20.742137 27351 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 277, Time since last communication: 0.000s
I20240501 14:00:20.750603 25112 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.755059 25247 raft_consensus.cc:1203] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.277->[1.278-1.278] Dedup: 1.278->[]
W20240501 14:00:20.766067 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 114907us
I20240501 14:00:20.767208 27402 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.775763 24981 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.777793 24588 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.783339 24596 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 275 to 279, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.785899 24450 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:20.817286 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 50828us
I20240501 14:00:20.822312 26214 raft_consensus.cc:1203] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.254->[1.255-1.256] Dedup: 1.255->[1.256-1.256]
I20240501 14:00:20.871882 25244 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.934491 27511 tablet_replica.cc:330] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: stopping tablet replica
I20240501 14:00:20.934777 27511 raft_consensus.cc:2229] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:20.959051 27511 pending_rounds.cc:62] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Trying to abort 1 pending ops.
I20240501 14:00:20.959288 27511 pending_rounds.cc:68] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Aborting op as it isn't in flight: id { term: 1 index: 277 } timestamp: 7022886996285222912 op_type: WRITE_OP write_request { tablet_id: "9762584f4ce94ab68a52bc1f446e569c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\205\204N\030<bQF\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c523779ab70285d8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022886996228337664 authz_token { token_data: "\010\377\227\311\261\006\"3\n\005slave\022*\n 014f9343c5e744958db9b1d185f88390\020\001\030\001 \001(\001" signature: "<redacted>""o\032:\343\354K\266\324\303\322\307\224\037\274\265\210\253H^\331Uu\0047*\021\356\314\324\274)\23279\017\240\376\305\222\301\322\354\320I\'\301(\230d\020\026\273G\305\002h7\327\374S\207\270Z\031" signing_key_seq_num: 0 } } request_id { client_id: "51d46c18bf444c6f99c8e1f88044984e" seq_no: 927 first_incomplete_seq_no: 927 attempt_no: 0 }
I20240501 14:00:20.961582 27511 raft_consensus.cc:2258] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:20.964496 25110 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:20.966281 27511 ts_tablet_manager.cc:1875] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:20.983163 24723 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:20.986654 24987 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240501 14:00:20.989207 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240501 14:00:21.004047 26216 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 203 to 256, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:21.044135 27511 ts_tablet_manager.cc:1888] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.277
I20240501 14:00:21.044358 27511 log.cc:1196] T 9762584f4ce94ab68a52bc1f446e569c P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:21.046650 24439 catalog_manager.cc:4830] TS 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): tablet 9762584f4ce94ab68a52bc1f446e569c (table rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]) successfully deleted
I20240501 14:00:21.070776 25246 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240501 14:00:21.071589 25116 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240501 14:00:21.114817 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240501 14:00:21.164286 26211 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240501 14:00:21.218353 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240501 14:00:21.272801 27331 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240501 14:00:21.351744 26891 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:21.379829 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0a8a92803e1459ca3338c8054b2f8b6 with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:21.388484 24452 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:21.389765 24987 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:21.455960 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 104639us
I20240501 14:00:21.463553 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 with delete_type TABLET_DATA_TOMBSTONED (TS 283529d70ba14547b9440a16350b629e not found in new config with opid_index 258) from {username='slave'} at 127.0.0.1:44100
I20240501 14:00:21.476428 25110 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:21.485724 27541 tablet_replica.cc:330] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: stopping tablet replica
I20240501 14:00:21.486011 27541 raft_consensus.cc:2229] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:21.496273 27541 pending_rounds.cc:62] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Trying to abort 1 pending ops.
I20240501 14:00:21.496452 27541 pending_rounds.cc:68] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Aborting op as it isn't in flight: id { term: 1 index: 257 } timestamp: 7022886997682241536 op_type: WRITE_OP write_request { tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\360\"\370lV\251x\020\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""e401c638e0f973a6" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022886997613694976 authz_token { token_data: "\010\201\230\311\261\006\"3\n\005slave\022*\n a722443079e943e28774847bc8347ec3\020\001\030\001 \001(\001" signature: "<redacted>""7\306\300\'\355\311\t\355\234\321-P\344\353\251#\030\244JH7\337\254)\372,\014L\3274\304\327\342\322^Uv\362\373\266\013\245*S\217i \373\321\220|m=\\\341Gh8\320\366q\347S\251" signing_key_seq_num: 0 } } request_id { client_id: "c378ed2be03f44f290962500d4ce3efe" seq_no: 806 first_incomplete_seq_no: 806 attempt_no: 0 }
I20240501 14:00:21.497092 27541 raft_consensus.cc:2258] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:21.502589 27541 ts_tablet_manager.cc:1875] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:21.522014 26211 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:21.529414 27541 ts_tablet_manager.cc:1888] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.257
I20240501 14:00:21.529640 27541 log.cc:1196] T c0a8a92803e1459ca3338c8054b2f8b6 P 283529d70ba14547b9440a16350b629e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:00:21.532078 24439 catalog_manager.cc:4830] TS 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): tablet c0a8a92803e1459ca3338c8054b2f8b6 (table rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]) successfully deleted
I20240501 14:00:21.539862 25247 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 256 to 258, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:21.577958 27316 log.cc:924] Time spent T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Append to log took a long time: real 0.236s user 0.003s sys 0.000s
W20240501 14:00:21.804121 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 176672us
W20240501 14:00:21.911242 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 72797.7us
I20240501 14:00:21.975008 27316 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap replayed 1/1 log segments. Stats: ops{read=221 overwritten=0 applied=220 ignored=0} inserts{seen=905 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:21.982748 27316 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Bootstrap complete.
I20240501 14:00:21.984284 27316 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Time spent bootstrapping tablet: real 6.292s user 0.295s sys 0.026s
I20240501 14:00:21.985044 27316 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:21.985688 27316 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f6d2177839a452eba5e2fc39cb40881, State: Initialized, Role: LEARNER
I20240501 14:00:21.985926 27316 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 220, Last appended: 1.221, Last appended by leader: 221, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: true } }
I20240501 14:00:21.995015 27316 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:00:22.002301 25137 tablet_copy_service.cc:342] P 491e21cc623d4879b09975ede207ab2d: Request end of tablet copy session 2f6d2177839a452eba5e2fc39cb40881-f64ab2203c9042989dbcab79bab164a6 received from {username='slave'} at 127.22.5.199:54893
I20240501 14:00:22.002450 25137 tablet_copy_service.cc:433] P 491e21cc623d4879b09975ede207ab2d: ending tablet copy session 2f6d2177839a452eba5e2fc39cb40881-f64ab2203c9042989dbcab79bab164a6 on tablet f64ab2203c9042989dbcab79bab164a6 with peer 2f6d2177839a452eba5e2fc39cb40881
W20240501 14:00:22.114365 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 198338us
I20240501 14:00:22.130407 26215 raft_consensus.cc:1203] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.212->[1.213-1.243] Dedup: 1.221->[1.222-1.243]
W20240501 14:00:22.348608 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 55763.5us
I20240501 14:00:22.455570 27444 raft_consensus.cc:1050] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: attempting to promote NON_VOTER 2f6d2177839a452eba5e2fc39cb40881 to VOTER
I20240501 14:00:22.457100 27444 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:22.460886 24596 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:00:22.461540 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 14:00:22.474681 26215 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 LEARNER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:00:22.475222 24853 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:00:22.481767 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 14:00:22.483146 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 14:00:22.506903 24987 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:00:22.507611 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 14:00:22.508872 24725 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240501 14:00:22.516405 27310 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 14:00:22.528982 24723 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.549546 26218 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.555907 24453 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 reported cstate change: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.557305 24984 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.630513 27445 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.679170 24596 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:22.710477 24856 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 213 to 245, 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:22.953905 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 48600us
I20240501 14:00:23.075646 25110 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:00:23.079134 24984 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 14:00:23.079816 27589 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 14:00:23.084539 24596 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 14:00:23.085197 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 14:00:23.092021 24853 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 14:00:23.097790 27444 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:23.105413 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f64ab2203c9042989dbcab79bab164a6 with cas_config_opid_index 245: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:23.106632 24981 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:23.107265 24450 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:23.109675 24591 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:23.127233 27472 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 14:00:23.134343 24852 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
W20240501 14:00:23.149092 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 80400.4us
I20240501 14:00:23.153859 26211 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 14:00:23.249873 27445 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.006s
I20240501 14:00:23.341694 26211 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 245 to 248, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } }
I20240501 14:00:23.398247 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet f64ab2203c9042989dbcab79bab164a6 with delete_type TABLET_DATA_TOMBSTONED (TS 909bf3bda47b4c7ab369f460f5e96973 not found in new config with opid_index 248) from {username='slave'} at 127.0.0.1:51724
I20240501 14:00:23.486277 27607 tablet_replica.cc:330] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: stopping tablet replica
I20240501 14:00:23.494959 27607 raft_consensus.cc:2229] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:23.518260 27607 raft_consensus.cc:2258] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:23.529124 27607 ts_tablet_manager.cc:1875] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:23.642201 27607 ts_tablet_manager.cc:1888] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.247
I20240501 14:00:23.654781 27607 log.cc:1196] T f64ab2203c9042989dbcab79bab164a6 P 909bf3bda47b4c7ab369f460f5e96973: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/f64ab2203c9042989dbcab79bab164a6
I20240501 14:00:23.664424 24441 catalog_manager.cc:4830] TS 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): tablet f64ab2203c9042989dbcab79bab164a6 (table rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]) successfully deleted
W20240501 14:00:23.997473 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 60830.4us
W20240501 14:00:24.160367 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 109093us
W20240501 14:00:24.429486 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 129993us
W20240501 14:00:24.551003 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 111600us
W20240501 14:00:24.561107 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 44849.3us
I20240501 14:00:25.497272 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:25.525316 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:25.544200 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:25.544221 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:25.585216 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:25.594578 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:25.599375 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 14:00:25.839567 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 96263us
W20240501 14:00:26.334337 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 101513us
W20240501 14:00:26.894685 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 140613us
I20240501 14:00:26.903679 24856 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.358, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } }
I20240501 14:00:26.908344 24595 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
W20240501 14:00:26.909399 24792 consensus_peers.cc:485] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b -> Peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Couldn't send request to peer 909bf3bda47b4c7ab369f460f5e96973. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9762584f4ce94ab68a52bc1f446e569c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:26.910105 26212 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 14:00:26.911037 27558 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 14:00:26.913874 25112 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 14:00:26.915194 27642 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 14:00:26.919973 27558 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 14:00:26.920783 25244 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 14:00:26.929353 27642 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 357, Time since last communication: 0.000s
W20240501 14:00:26.930928 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 45913us
I20240501 14:00:26.949716 24594 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } } }
I20240501 14:00:26.937383 27729 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.359: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } } }
I20240501 14:00:26.968101 25243 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } } }
I20240501 14:00:26.971297 26217 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } } }
I20240501 14:00:26.985881 24448 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:27.101197 25112 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 279 to 359, NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } } }
W20240501 14:00:27.263661 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 46463.1us
W20240501 14:00:27.335331 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 60210.5us
I20240501 14:00:27.591739 27755 ts_tablet_manager.cc:902] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Initiating tablet copy from peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
W20240501 14:00:27.633180 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 48048.5us
I20240501 14:00:27.729034 27755 tablet_copy_client.cc:308] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:37091
I20240501 14:00:27.823390 24877 tablet_copy_service.cc:140] P f74cb76209d04397a8a206f86e91ac5b: Received BeginTabletCopySession request for tablet 9762584f4ce94ab68a52bc1f446e569c from peer 909bf3bda47b4c7ab369f460f5e96973 ({username='slave'} at 127.22.5.194:42105)
I20240501 14:00:27.823555 24877 tablet_copy_service.cc:161] P f74cb76209d04397a8a206f86e91ac5b: Beginning new tablet copy session on tablet 9762584f4ce94ab68a52bc1f446e569c from peer 909bf3bda47b4c7ab369f460f5e96973 at {username='slave'} at 127.22.5.194:42105: session id = 909bf3bda47b4c7ab369f460f5e96973-9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:27.827122 24877 tablet_copy_source_session.cc:205] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:27.892328 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9762584f4ce94ab68a52bc1f446e569c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.903913 27755 tablet_copy_client.cc:783] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:27.904353 27755 tablet_copy_client.cc:647] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: tablet copy: Starting download of 1 WAL segments...
W20240501 14:00:27.923223 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 86239.3us
I20240501 14:00:27.974876 27755 tablet_copy_client.cc:523] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:27.979869 27755 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap starting.
W20240501 14:00:28.616987 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 51730.2us
W20240501 14:00:28.872388 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 44085.4us
W20240501 14:00:28.904680 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 133373us
W20240501 14:00:28.907466 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 115012us
W20240501 14:00:28.957645 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 50410us
W20240501 14:00:28.999164 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 94139us
I20240501 14:00:29.159196 27755 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap replayed 1/1 log segments. Stats: ops{read=367 overwritten=0 applied=366 ignored=0} inserts{seen=1048 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:29.159888 27755 tablet_bootstrap.cc:492] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Bootstrap complete.
I20240501 14:00:29.160331 27755 ts_tablet_manager.cc:1367] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Time spent bootstrapping tablet: real 1.181s user 0.420s sys 0.032s
I20240501 14:00:29.161561 27755 raft_consensus.cc:348] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } }
I20240501 14:00:29.162654 27755 raft_consensus.cc:729] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 909bf3bda47b4c7ab369f460f5e96973, State: Initialized, Role: LEARNER
I20240501 14:00:29.162932 27755 consensus_queue.cc:260] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 366, Last appended: 1.367, Last appended by leader: 367, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: true } }
I20240501 14:00:29.163692 27755 ts_tablet_manager.cc:1398] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 14:00:29.179317 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 108048us
I20240501 14:00:29.250020 24877 tablet_copy_service.cc:342] P f74cb76209d04397a8a206f86e91ac5b: Request end of tablet copy session 909bf3bda47b4c7ab369f460f5e96973-9762584f4ce94ab68a52bc1f446e569c received from {username='slave'} at 127.22.5.194:42105
I20240501 14:00:29.250180 24877 tablet_copy_service.cc:433] P f74cb76209d04397a8a206f86e91ac5b: ending tablet copy session 909bf3bda47b4c7ab369f460f5e96973-9762584f4ce94ab68a52bc1f446e569c on tablet 9762584f4ce94ab68a52bc1f446e569c with peer 909bf3bda47b4c7ab369f460f5e96973
I20240501 14:00:29.404930 24724 raft_consensus.cc:1203] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.358->[1.359-1.387] Dedup: 1.367->[1.368-1.387]
I20240501 14:00:29.558164 27729 raft_consensus.cc:1050] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b: attempting to promote NON_VOTER 909bf3bda47b4c7ab369f460f5e96973 to VOTER
I20240501 14:00:29.559115 27729 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 1.388, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } }
I20240501 14:00:29.564195 24723 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEARNER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240501 14:00:29.564224 26212 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240501 14:00:29.564319 25246 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240501 14:00:29.564725 27729 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240501 14:00:29.565506 27783 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240501 14:00:29.566345 27783 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240501 14:00:29.569448 24594 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240501 14:00:29.570039 27729 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240501 14:00:29.593672 27729 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:29.599107 26218 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:29.601292 24594 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:29.600848 24450 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:29.599637 25247 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:29.624017 24723 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
W20240501 14:00:29.654028 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 115625us
I20240501 14:00:29.679903 25116 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240501 14:00:29.718206 27558 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 387, Time since last communication: 0.000s
I20240501 14:00:29.824368 25111 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 359 to 389, 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:30.152235 24854 consensus_queue.cc:237] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 1.395, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } }
I20240501 14:00:30.174293 26211 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240501 14:00:30.174810 24726 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240501 14:00:30.175410 27551 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240501 14:00:30.178776 25243 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240501 14:00:30.180383 27551 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 394, Time since last communication: 0.000s
I20240501 14:00:30.180868 27551 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240501 14:00:30.184197 27652 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.396: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:30.184859 26211 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
W20240501 14:00:30.186667 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 59471us
I20240501 14:00:30.189890 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9762584f4ce94ab68a52bc1f446e569c with cas_config_opid_index 389: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:30.192188 24723 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:30.194700 25243 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
I20240501 14:00:30.198544 24448 catalog_manager.cc:5484] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:30.215158 25110 raft_consensus.cc:1261] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240501 14:00:30.217743 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet 9762584f4ce94ab68a52bc1f446e569c with delete_type TABLET_DATA_TOMBSTONED (TS 283529d70ba14547b9440a16350b629e not found in new config with opid_index 396) from {username='slave'} at 127.0.0.1:44100
I20240501 14:00:30.257035 27858 tablet_replica.cc:330] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: stopping tablet replica
W20240501 14:00:30.260206 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 164856us
I20240501 14:00:30.260672 27858 raft_consensus.cc:2229] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:30.261104 27858 raft_consensus.cc:2258] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:30.266583 27858 ts_tablet_manager.cc:1875] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:30.311544 27551 consensus_queue.cc:1035] T 9762584f4ce94ab68a52bc1f446e569c P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240501 14:00:30.328310 27858 ts_tablet_manager.cc:1888] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.395
I20240501 14:00:30.328447 27858 log.cc:1196] T 9762584f4ce94ab68a52bc1f446e569c P 283529d70ba14547b9440a16350b629e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:30.329263 24439 catalog_manager.cc:4830] TS 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): tablet 9762584f4ce94ab68a52bc1f446e569c (table rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]) successfully deleted
I20240501 14:00:30.414110 25112 raft_consensus.cc:2932] T 9762584f4ce94ab68a52bc1f446e569c P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 389 to 396, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { promote: false } } }
W20240501 14:00:30.869633 25050 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 43729.2us
W20240501 14:00:30.918578 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 55388.5us
W20240501 14:00:30.918006 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 55277.6us
W20240501 14:00:30.982489 25180 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 104983us
W20240501 14:00:31.054809 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 70944.5us
W20240501 14:00:31.281625 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 74300.4us
W20240501 14:00:31.569840 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 45726.9us
I20240501 14:00:32.125392 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:32.131525 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:32.197120 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:32.329474 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:32.335595 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:32.344431 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:00:32.431785 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 49004us
I20240501 14:00:32.511160 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:32.537016 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 87269.8us
W20240501 14:00:32.690142 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 80196.1us
W20240501 14:00:33.630582 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 61680.3us
W20240501 14:00:34.068571 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 51485.9us
W20240501 14:00:34.089607 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 70340.6us
W20240501 14:00:34.730515 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 150376us
W20240501 14:00:35.142872 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 57235.1us
I20240501 14:00:35.288820 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:35.289497 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:00:35.302009 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 44738.2us
I20240501 14:00:35.338951 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:35.355736 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:35.363750 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:35.398356 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:00:35.621624 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 48953.3us
W20240501 14:00:35.834625 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 98334.7us
W20240501 14:00:35.855829 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 54874.2us
I20240501 14:00:35.873950 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:00:35.927165 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 112949us
W20240501 14:00:35.927629 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 54765.7us
W20240501 14:00:36.158218 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 74173.2us
W20240501 14:00:36.657685 24947 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.106s sys 0.255s
W20240501 14:00:36.657920 24947 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.517s user 0.106s sys 0.255s
W20240501 14:00:36.660298 24985 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.006s sys 0.001s
W20240501 14:00:36.660403 24985 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.516s user 0.006s sys 0.001s
W20240501 14:00:37.161783 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 70053.9us
W20240501 14:00:37.825737 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 144290us
W20240501 14:00:38.020453 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 82818.5us
W20240501 14:00:38.589097 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 54845.7us
W20240501 14:00:38.981186 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 45820.6us
W20240501 14:00:39.696316 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 107441us
W20240501 14:00:39.779857 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 50123.2us
W20240501 14:00:39.856575 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 70873.8us
W20240501 14:00:40.063619 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 44644.6us
W20240501 14:00:40.172221 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 57658us
W20240501 14:00:40.221385 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 62504.6us
W20240501 14:00:40.505328 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 87639.8us
W20240501 14:00:40.557178 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 61259.7us
W20240501 14:00:41.361325 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 88391.5us
W20240501 14:00:41.552269 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 140133us
W20240501 14:00:42.348446 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 185012us
W20240501 14:00:42.625108 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 275816us
W20240501 14:00:42.890976 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 61199.9us
W20240501 14:00:43.027814 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 132149us
W20240501 14:00:43.220330 24440 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.002s sys 0.018s
W20240501 14:00:43.220572 24440 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.517s user 0.002s sys 0.018s
W20240501 14:00:43.543694 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 86995.8us
W20240501 14:00:43.982416 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 75706.8us
W20240501 14:00:44.315955 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 106950us
W20240501 14:00:44.676535 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 76399.8us
W20240501 14:00:44.709455 28233 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.061s sys 0.073s
W20240501 14:00:44.709674 28233 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.061s sys 0.073s
W20240501 14:00:45.075882 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 60574.2us
W20240501 14:00:45.148711 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 43739.5us
I20240501 14:00:45.196173 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:45.330621 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:45.350234 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:00:45.376828 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 52925.4us
I20240501 14:00:45.411208 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:45.442103 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:45.492633 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:45.507862 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 82873.6us
I20240501 14:00:45.850800 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:00:46.006058 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 112387us
W20240501 14:00:46.006336 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 112757us
W20240501 14:00:46.298938 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 85272.5us
W20240501 14:00:46.453697 24532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 55073.1us
W20240501 14:00:46.454396 24793 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 55502.8us
W20240501 14:00:46.598836 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 53637.3us
W20240501 14:00:46.953630 24922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:45987, user_credentials={real_user=slave}} blocked reactor thread for 72384.4us
W20240501 14:00:47.345029 25053 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:35589, user_credentials={real_user=slave}} blocked reactor thread for 75041.5us
W20240501 14:00:48.025851 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 77521.5us
W20240501 14:00:48.258482 24923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:37091, user_credentials={real_user=slave}} blocked reactor thread for 58643.1us
I20240501 14:00:48.307091 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:48.344866 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:48.418201 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:48.440690 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:48.472256 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:48.543823 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:00:48.649974 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 67569.4us
W20240501 14:00:48.783188 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 82494.8us
W20240501 14:00:48.838485 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 125400us
W20240501 14:00:48.845716 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 57319.3us
W20240501 14:00:48.882211 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 49809.9us
W20240501 14:00:48.944416 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 59152.9us
W20240501 14:00:49.062964 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 61008.6us
W20240501 14:00:49.064646 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 61273.2us
W20240501 14:00:49.157387 24660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35547, user_credentials={real_user=slave}} blocked reactor thread for 51133.6us
W20240501 14:00:49.156833 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:35069, user_credentials={real_user=slave}} blocked reactor thread for 50419.3us
W20240501 14:00:49.158569 24790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 51788.6us
I20240501 14:00:49.241389 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 14:00:49.633525 24663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:32877, user_credentials={real_user=slave}} blocked reactor thread for 59813.8us
W20240501 14:00:49.770916 25242 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.097s sys 0.095s
W20240501 14:00:49.771128 25242 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.524s user 0.097s sys 0.096s
W20240501 14:00:49.854192 24595 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.028s sys 0.033s
W20240501 14:00:49.862814 24595 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.569s user 0.028s sys 0.033s
W20240501 14:00:50.314132 25050 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:43227, user_credentials={real_user=slave}} blocked reactor thread for 108373us
I20240501 14:00:51.409837 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:51.421226 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:51.440038 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:51.443074 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:51.443739 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:51.453737 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:51.468361 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
5c4f17b3dc294f979e57abb381a3996f | 127.22.5.254:40379 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:34597 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
283529d70ba14547b9440a16350b629e | 127.22.5.193:35547 | HEALTHY | <none> | 2 | 0
2f6d2177839a452eba5e2fc39cb40881 | 127.22.5.199:35589 | HEALTHY | <none> | 0 | 0
491e21cc623d4879b09975ede207ab2d | 127.22.5.197:35069 | HEALTHY | <none> | 2 | 0
4e03e0d09b974c7d8cf7f49652d4fd23 | 127.22.5.196:43227 | HEALTHY | <none> | 2 | 0
84c5917a503d4a82bb2d9c0d5fdb04bd | 127.22.5.198:32877 | HEALTHY | <none> | 2 | 0
909bf3bda47b4c7ab369f460f5e96973 | 127.22.5.194:45987 | HEALTHY | <none> | 5 | 0
f74cb76209d04397a8a206f86e91ac5b | 127.22.5.195:37091 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.5.193 | experimental | 127.22.5.193:35547
local_ip_for_outbound_sockets | 127.22.5.194 | experimental | 127.22.5.194:45987
local_ip_for_outbound_sockets | 127.22.5.195 | experimental | 127.22.5.195:37091
local_ip_for_outbound_sockets | 127.22.5.196 | experimental | 127.22.5.196:43227
local_ip_for_outbound_sockets | 127.22.5.197 | experimental | 127.22.5.197:35069
local_ip_for_outbound_sockets | 127.22.5.198 | experimental | 127.22.5.198:32877
local_ip_for_outbound_sockets | 127.22.5.199 | experimental | 127.22.5.199:35589
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.5.193:35547
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.5.194:45987
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.5.195:37091
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.5.196:43227
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.5.197:35069
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.5.198:32877
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.5.199:35589
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:34597 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:00:51.583869 22551 log_verifier.cc:126] Checking tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:00:52.075572 22551 log_verifier.cc:177] Verified matching terms for 318 ops in tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:00:52.076562 22551 log_verifier.cc:126] Checking tablet 2dbc87150c2745c3854334dad7b7ed75
I20240501 14:00:52.401700 22551 log_verifier.cc:177] Verified matching terms for 315 ops in tablet 2dbc87150c2745c3854334dad7b7ed75
I20240501 14:00:52.402523 22551 log_verifier.cc:126] Checking tablet 2f1344fe8e4742e5914f49ffcaba21f8
I20240501 14:00:52.813333 22551 log_verifier.cc:177] Verified matching terms for 417 ops in tablet 2f1344fe8e4742e5914f49ffcaba21f8
I20240501 14:00:52.814791 22551 log_verifier.cc:126] Checking tablet 3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:00:53.418751 22551 log_verifier.cc:177] Verified matching terms for 666 ops in tablet 3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:00:53.420179 22551 log_verifier.cc:126] Checking tablet 94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:00:53.858144 22551 log_verifier.cc:177] Verified matching terms for 445 ops in tablet 94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:00:53.859248 22551 log_verifier.cc:126] Checking tablet 96a1c9e9695f4e0cbab9d05138eef65a
I20240501 14:00:54.555688 22551 log_verifier.cc:177] Verified matching terms for 649 ops in tablet 96a1c9e9695f4e0cbab9d05138eef65a
I20240501 14:00:54.557228 22551 log_verifier.cc:126] Checking tablet 9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:55.129400 22551 log_verifier.cc:177] Verified matching terms for 620 ops in tablet 9762584f4ce94ab68a52bc1f446e569c
I20240501 14:00:55.130784 22551 log_verifier.cc:126] Checking tablet 9ed1065df4344515b0f598f912e22149
I20240501 14:00:55.486413 22551 log_verifier.cc:177] Verified matching terms for 362 ops in tablet 9ed1065df4344515b0f598f912e22149
I20240501 14:00:55.487507 22551 log_verifier.cc:126] Checking tablet bb4b73ec3b4146fda1248c7cb800afbd
I20240501 14:00:55.929639 22551 log_verifier.cc:177] Verified matching terms for 467 ops in tablet bb4b73ec3b4146fda1248c7cb800afbd
I20240501 14:00:55.931030 22551 log_verifier.cc:126] Checking tablet c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:00:56.365485 22551 log_verifier.cc:177] Verified matching terms for 447 ops in tablet c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:00:56.366560 22551 log_verifier.cc:126] Checking tablet c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:00:56.806253 22551 log_verifier.cc:177] Verified matching terms for 442 ops in tablet c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:00:56.807243 22551 log_verifier.cc:126] Checking tablet d02fe94840e24205aa98b3190d089784
I20240501 14:00:57.259927 22551 log_verifier.cc:177] Verified matching terms for 435 ops in tablet d02fe94840e24205aa98b3190d089784
I20240501 14:00:57.261027 22551 log_verifier.cc:126] Checking tablet d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:00:57.308068 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:00:57.308183 22551 log_verifier.cc:126] Checking tablet dfe9cb33f20b45aa90b066e210804355
I20240501 14:00:57.733450 22551 log_verifier.cc:177] Verified matching terms for 459 ops in tablet dfe9cb33f20b45aa90b066e210804355
I20240501 14:00:57.734716 22551 log_verifier.cc:126] Checking tablet f64ab2203c9042989dbcab79bab164a6
I20240501 14:00:58.132006 22551 log_verifier.cc:177] Verified matching terms for 399 ops in tablet f64ab2203c9042989dbcab79bab164a6
I20240501 14:00:58.132992 22551 log_verifier.cc:126] Checking tablet f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:00:58.541822 22551 log_verifier.cc:177] Verified matching terms for 412 ops in tablet f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:00:58.556375 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.200:0
--local_ip_for_outbound_sockets=127.22.5.200
--webserver_interface=127.22.5.200
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--builtin_ntp_servers=127.22.5.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:00:58.951759 28530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:58.952073 28530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:58.952214 28530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:58.961656 28530 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:58.961815 28530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:58.962023 28530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.200
I20240501 14:00:58.972433 28530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.5.200
--webserver_port=0
--tserver_master_addrs=127.22.5.254:40379
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.200
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:58.973760 28530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:58.975541 28530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:58.990764 28538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:58.991384 28540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:58.991858 28537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:58.992591 28530 server_base.cc:1034] running on GCE node
I20240501 14:00:58.993263 28530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:58.994231 28530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:58.995437 28530 hybrid_clock.cc:648] HybridClock initialized: now 1714572058995389 us; error 39 us; skew 500 ppm
I20240501 14:00:58.995797 28530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:58.998510 28530 webserver.cc:460] Webserver started at http://127.22.5.200:44961/ using document root <none> and password file <none>
I20240501 14:00:58.999308 28530 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:58.999418 28530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:58.999768 28530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:59.002956 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c82fcb40e5534742981e8300df76a1f7"
format_stamp: "Formatted at 2024-05-01 14:00:58 on dist-test-slave-5jzj"
I20240501 14:00:59.003749 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c82fcb40e5534742981e8300df76a1f7"
format_stamp: "Formatted at 2024-05-01 14:00:58 on dist-test-slave-5jzj"
I20240501 14:00:59.009292 28530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240501 14:00:59.013098 28546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:59.015230 28530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:59.015399 28530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c82fcb40e5534742981e8300df76a1f7"
format_stamp: "Formatted at 2024-05-01 14:00:58 on dist-test-slave-5jzj"
I20240501 14:00:59.015607 28530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:59.034624 28530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:59.035542 28530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:59.035856 28530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:59.037082 28530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:59.042759 28530 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:59.042876 28530 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:59.043021 28530 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:59.043134 28530 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:59.091838 28530 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.200:37067
I20240501 14:00:59.091907 28661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.200:37067 every 8 connection(s)
I20240501 14:00:59.093255 28530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:00:59.101538 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 28530
I20240501 14:00:59.101756 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:00:59.109823 28662 heartbeater.cc:347] Connected to a master server at 127.22.5.254:40379
I20240501 14:00:59.110184 28662 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:59.111133 28662 heartbeater.cc:511] Master 127.22.5.254:40379 requested a full tablet report, sending...
I20240501 14:00:59.113018 24452 ts_manager.cc:194] Registered new tserver with Master: c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:00:59.113996 24452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.200:52979
I20240501 14:00:59.675073 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:59.706986 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:59.714949 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:59.738739 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:59.751222 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:59.775316 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:59.792462 28597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:59.813977 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:00.007553 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:00.022498 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:00.029857 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:00.035174 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:00.052623 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:00.055269 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:00.071089 28597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:00.080827 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:00.116782 28662 heartbeater.cc:503] Master 127.22.5.254:40379 was elected leader, sending a full tablet report...
I20240501 14:01:00.192238 25242 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 459, Committed index: 459, Last appended: 1.459, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:00.195574 24987 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 459. Preceding OpId from leader: term: 1 index: 460. (index mismatch)
I20240501 14:01:00.196301 24595 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 459. Preceding OpId from leader: term: 1 index: 460. (index mismatch)
I20240501 14:01:00.197103 24724 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 459. Preceding OpId from leader: term: 1 index: 460. (index mismatch)
I20240501 14:01:00.197651 28251 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 459, Time since last communication: 0.000s
I20240501 14:01:00.198072 28759 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 459, Time since last communication: 0.000s
I20240501 14:01:00.198411 28760 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 459, Time since last communication: 0.000s
I20240501 14:01:00.208927 25114 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 459. Preceding OpId from leader: term: 1 index: 460. (index mismatch)
I20240501 14:01:00.210400 28759 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 459, Time since last communication: 0.000s
W20240501 14:01:00.219805 25183 consensus_peers.cc:485] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfe9cb33f20b45aa90b066e210804355. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.219974 28760 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.460: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.220839 24595 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.460: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.225101 24987 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.460: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.227108 24447 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.231586 24724 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.460: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.234849 25114 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.460: config changed from index 96 to 460, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.245792 24595 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 445, Committed index: 445, Last appended: 1.445, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:00.250229 24724 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240501 14:01:00.250229 24854 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240501 14:01:00.250260 26218 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240501 14:01:00.250870 28777 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240501 14:01:00.251340 24987 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240501 14:01:00.251271 28776 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240501 14:01:00.251585 28125 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240501 14:01:00.252097 28778 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240501 14:01:00.266985 28778 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.446: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.268455 24724 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.268421 24854 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.273990 24987 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.277015 24454 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.276893 26218 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 169 to 446, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
W20240501 14:01:00.282204 24532 consensus_peers.cc:485] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94749eeb6ba44d3ea0803cacf47017ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.301277 24854 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:00.305570 24987 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20240501 14:01:00.306290 28791 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240501 14:01:00.306934 26218 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20240501 14:01:00.309561 28564 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240501 14:01:00.314527 28564 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.319226 26218 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.325682 24454 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 reported cstate change: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.328632 25242 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20240501 14:01:00.328840 24595 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
W20240501 14:01:00.329988 24793 consensus_peers.cc:485] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f7c6d19fc7402ebc9fa3a3f07f76e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.331102 24987 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.334772 28564 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240501 14:01:00.335376 28564 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240501 14:01:00.341081 24595 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.348505 25242 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 144 to 319, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.352375 25114 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:00.358583 26218 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20240501 14:01:00.359313 24854 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20240501 14:01:00.360548 28806 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240501 14:01:00.360913 28524 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
W20240501 14:01:00.367091 25053 consensus_peers.cc:485] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f64ab2203c9042989dbcab79bab164a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.370298 24595 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20240501 14:01:00.370563 24987 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20240501 14:01:00.370873 28524 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240501 14:01:00.371538 28524 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.400: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.373064 28806 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240501 14:01:00.375762 24595 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.380918 24987 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.381489 24854 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.382506 24448 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.390349 26218 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 248 to 400, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.409941 24987 consensus_queue.cc:237] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 666, Committed index: 666, Last appended: 1.666, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:00.413271 24724 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 666. Preceding OpId from leader: term: 1 index: 667. (index mismatch)
I20240501 14:01:00.413569 24595 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 666. Preceding OpId from leader: term: 1 index: 667. (index mismatch)
I20240501 14:01:00.413626 24854 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 666. Preceding OpId from leader: term: 1 index: 667. (index mismatch)
I20240501 14:01:00.414162 28712 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 667, Last known committed idx: 666, Time since last communication: 0.000s
I20240501 14:01:00.414703 28712 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 667, Last known committed idx: 666, Time since last communication: 0.000s
I20240501 14:01:00.415174 28712 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 667, Last known committed idx: 666, Time since last communication: 0.000s
I20240501 14:01:00.416632 25114 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 666. Preceding OpId from leader: term: 1 index: 667. (index mismatch)
I20240501 14:01:00.417330 28712 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 667, Last known committed idx: 666, Time since last communication: 0.000s
I20240501 14:01:00.418817 28821 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.667: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
W20240501 14:01:00.420392 24923 consensus_peers.cc:485] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dfc84e5be47406f95b6246252e8d0f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.420892 24854 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.667: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.421993 25114 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.667: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.423177 24724 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.667: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.427364 24452 catalog_manager.cc:5484] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.430857 24595 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.667: config changed from index -1 to 667, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:00.618582 28852 ts_tablet_manager.cc:902] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 14:01:00.620364 28852 tablet_copy_client.cc:308] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:32877
I20240501 14:01:00.626849 25267 tablet_copy_service.cc:140] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Received BeginTabletCopySession request for tablet dfe9cb33f20b45aa90b066e210804355 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:45049)
I20240501 14:01:00.626997 25267 tablet_copy_service.cc:161] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Beginning new tablet copy session on tablet dfe9cb33f20b45aa90b066e210804355 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:45049: session id = c82fcb40e5534742981e8300df76a1f7-dfe9cb33f20b45aa90b066e210804355
I20240501 14:01:00.629012 25267 tablet_copy_source_session.cc:205] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:00.633844 28852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe9cb33f20b45aa90b066e210804355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.644353 28852 tablet_copy_client.cc:783] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:00.644676 28852 tablet_copy_client.cc:647] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:00.648134 28852 tablet_copy_client.cc:523] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:00.651762 28852 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:00.724342 28852 log.cc:824] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:00.861971 28885 ts_tablet_manager.cc:902] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547)
I20240501 14:01:00.863713 28885 tablet_copy_client.cc:308] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35547
I20240501 14:01:00.870090 24616 tablet_copy_service.cc:140] P 283529d70ba14547b9440a16350b629e: Received BeginTabletCopySession request for tablet 94749eeb6ba44d3ea0803cacf47017ea from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:34751)
I20240501 14:01:00.870254 24616 tablet_copy_service.cc:161] P 283529d70ba14547b9440a16350b629e: Beginning new tablet copy session on tablet 94749eeb6ba44d3ea0803cacf47017ea from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:34751: session id = c82fcb40e5534742981e8300df76a1f7-94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:01:00.872146 24616 tablet_copy_source_session.cc:205] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:00.873564 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94749eeb6ba44d3ea0803cacf47017ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.879638 28890 ts_tablet_manager.cc:902] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 14:01:00.880601 28885 tablet_copy_client.cc:783] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:00.880952 28885 tablet_copy_client.cc:647] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:00.883055 28890 tablet_copy_client.cc:308] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:43227
I20240501 14:01:00.884766 28885 tablet_copy_client.cc:523] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:00.887442 28885 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:00.890962 25007 tablet_copy_service.cc:140] P 4e03e0d09b974c7d8cf7f49652d4fd23: Received BeginTabletCopySession request for tablet 3dfc84e5be47406f95b6246252e8d0f5 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:50315)
I20240501 14:01:00.891146 25007 tablet_copy_service.cc:161] P 4e03e0d09b974c7d8cf7f49652d4fd23: Beginning new tablet copy session on tablet 3dfc84e5be47406f95b6246252e8d0f5 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:50315: session id = c82fcb40e5534742981e8300df76a1f7-3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:01:00.893996 25007 tablet_copy_source_session.cc:205] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:00.895485 28890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfc84e5be47406f95b6246252e8d0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.904752 28890 tablet_copy_client.cc:783] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:00.905133 28890 tablet_copy_client.cc:647] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:00.910990 28890 tablet_copy_client.cc:523] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:00.912518 28895 ts_tablet_manager.cc:902] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 14:01:00.914754 28890 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:00.961370 28899 ts_tablet_manager.cc:902] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 14:01:00.972290 28895 tablet_copy_client.cc:308] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:37091
I20240501 14:01:00.981349 24877 tablet_copy_service.cc:140] P f74cb76209d04397a8a206f86e91ac5b: Received BeginTabletCopySession request for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:50913)
I20240501 14:01:00.981551 24877 tablet_copy_service.cc:161] P f74cb76209d04397a8a206f86e91ac5b: Beginning new tablet copy session on tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:50913: session id = c82fcb40e5534742981e8300df76a1f7-11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:01:00.984972 24877 tablet_copy_source_session.cc:205] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:00.993141 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.008144 28895 tablet_copy_client.cc:783] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.008787 28895 tablet_copy_client.cc:647] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.012558 28895 tablet_copy_client.cc:523] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:01.015324 28899 tablet_copy_client.cc:308] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:35069
I20240501 14:01:01.024020 25137 tablet_copy_service.cc:140] P 491e21cc623d4879b09975ede207ab2d: Received BeginTabletCopySession request for tablet f64ab2203c9042989dbcab79bab164a6 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:54763)
I20240501 14:01:01.024210 25137 tablet_copy_service.cc:161] P 491e21cc623d4879b09975ede207ab2d: Beginning new tablet copy session on tablet f64ab2203c9042989dbcab79bab164a6 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:54763: session id = c82fcb40e5534742981e8300df76a1f7-f64ab2203c9042989dbcab79bab164a6
I20240501 14:01:01.027487 28895 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:01.027526 25137 tablet_copy_source_session.cc:205] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.029420 28899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64ab2203c9042989dbcab79bab164a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.042582 28899 tablet_copy_client.cc:783] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.042966 28899 tablet_copy_client.cc:647] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.046716 28899 tablet_copy_client.cc:523] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:01.050511 28899 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:01.430222 28852 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=460 overwritten=0 applied=460 ignored=0} inserts{seen=1088 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:01.430915 28852 tablet_bootstrap.cc:492] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:01.432001 28852 ts_tablet_manager.cc:1367] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.780s user 0.510s sys 0.063s
I20240501 14:01:01.438812 28852 raft_consensus.cc:348] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.439414 28852 raft_consensus.cc:729] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:01.440099 28852 consensus_queue.cc:260] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 460, Last appended: 1.460, Last appended by leader: 460, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 460 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.449393 28852 ts_tablet_manager.cc:1398] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20240501 14:01:01.450325 25267 tablet_copy_service.cc:342] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-dfe9cb33f20b45aa90b066e210804355 received from {username='slave'} at 127.22.5.200:45049
I20240501 14:01:01.450479 25267 tablet_copy_service.cc:433] P 84c5917a503d4a82bb2d9c0d5fdb04bd: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-dfe9cb33f20b45aa90b066e210804355 on tablet dfe9cb33f20b45aa90b066e210804355 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:01.849694 28617 raft_consensus.cc:1203] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.459->[1.460-1.460] Dedup: 1.460->[]
I20240501 14:01:01.878531 28885 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=446 overwritten=0 applied=446 ignored=0} inserts{seen=1128 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:01.879050 28885 tablet_bootstrap.cc:492] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:01.879429 28885 ts_tablet_manager.cc:1367] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.992s user 0.572s sys 0.065s
I20240501 14:01:01.880748 28885 raft_consensus.cc:348] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.881065 28885 raft_consensus.cc:729] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:01.881320 28885 consensus_queue.cc:260] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 446, Last appended: 1.446, Last appended by leader: 446, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.892408 28885 ts_tablet_manager.cc:1398] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:01:01.893373 24616 tablet_copy_service.cc:342] P 283529d70ba14547b9440a16350b629e: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-94749eeb6ba44d3ea0803cacf47017ea received from {username='slave'} at 127.22.5.200:34751
I20240501 14:01:01.893522 24616 tablet_copy_service.cc:433] P 283529d70ba14547b9440a16350b629e: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-94749eeb6ba44d3ea0803cacf47017ea on tablet 94749eeb6ba44d3ea0803cacf47017ea with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:01.912691 28899 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=400 overwritten=0 applied=400 ignored=0} inserts{seen=1082 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:01.924188 28899 tablet_bootstrap.cc:492] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:01.924750 28899 ts_tablet_manager.cc:1367] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.874s user 0.448s sys 0.069s
I20240501 14:01:01.926324 28899 raft_consensus.cc:348] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.926654 28899 raft_consensus.cc:729] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:01.929425 28899 consensus_queue.cc:260] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 400, Last appended: 1.400, Last appended by leader: 400, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.930238 28899 ts_tablet_manager.cc:1398] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:01:01.931428 25137 tablet_copy_service.cc:342] P 491e21cc623d4879b09975ede207ab2d: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-f64ab2203c9042989dbcab79bab164a6 received from {username='slave'} at 127.22.5.200:54763
I20240501 14:01:01.931582 25137 tablet_copy_service.cc:433] P 491e21cc623d4879b09975ede207ab2d: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-f64ab2203c9042989dbcab79bab164a6 on tablet f64ab2203c9042989dbcab79bab164a6 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:01.958233 28895 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=319 overwritten=0 applied=319 ignored=0} inserts{seen=1001 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:01.958779 28895 tablet_bootstrap.cc:492] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:01.959144 28895 ts_tablet_manager.cc:1367] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.932s user 0.375s sys 0.060s
I20240501 14:01:01.960016 28895 raft_consensus.cc:348] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.960323 28895 raft_consensus.cc:729] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:01.960582 28895 consensus_queue.cc:260] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 319, Last appended: 1.319, Last appended by leader: 319, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:01.965031 28895 ts_tablet_manager.cc:1398] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:01.966689 24877 tablet_copy_service.cc:342] P f74cb76209d04397a8a206f86e91ac5b: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-11f7c6d19fc7402ebc9fa3a3f07f76e0 received from {username='slave'} at 127.22.5.200:50913
I20240501 14:01:01.966837 24877 tablet_copy_service.cc:433] P f74cb76209d04397a8a206f86e91ac5b: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-11f7c6d19fc7402ebc9fa3a3f07f76e0 on tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:01.985870 28617 raft_consensus.cc:1203] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.318->[1.319-1.319] Dedup: 1.319->[]
I20240501 14:01:02.050752 28617 raft_consensus.cc:1203] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.445->[1.446-1.446] Dedup: 1.446->[]
I20240501 14:01:02.127162 28890 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=667 overwritten=0 applied=667 ignored=0} inserts{seen=1339 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:02.127687 28890 tablet_bootstrap.cc:492] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:02.128121 28890 ts_tablet_manager.cc:1367] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 1.213s user 0.799s sys 0.065s
I20240501 14:01:02.129096 28890 raft_consensus.cc:348] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:02.129529 28890 raft_consensus.cc:729] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:02.129789 28890 consensus_queue.cc:260] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 667, Last appended: 1.667, Last appended by leader: 667, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 667 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:02.130582 28890 ts_tablet_manager.cc:1398] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 14:01:02.131659 25007 tablet_copy_service.cc:342] P 4e03e0d09b974c7d8cf7f49652d4fd23: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-3dfc84e5be47406f95b6246252e8d0f5 received from {username='slave'} at 127.22.5.200:50315
I20240501 14:01:02.131776 25007 tablet_copy_service.cc:433] P 4e03e0d09b974c7d8cf7f49652d4fd23: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-3dfc84e5be47406f95b6246252e8d0f5 on tablet 3dfc84e5be47406f95b6246252e8d0f5 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:02.331568 28951 raft_consensus.cc:1050] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:02.332466 28951 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 460, Committed index: 460, Last appended: 1.460, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.335234 24987 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 460. Preceding OpId from leader: term: 1 index: 461. (index mismatch)
I20240501 14:01:02.335269 24591 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 460. Preceding OpId from leader: term: 1 index: 461. (index mismatch)
I20240501 14:01:02.335341 28617 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 460. Preceding OpId from leader: term: 1 index: 461. (index mismatch)
I20240501 14:01:02.335881 28998 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I20240501 14:01:02.336416 28950 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I20240501 14:01:02.336699 28759 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I20240501 14:01:02.338004 25114 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 460. Preceding OpId from leader: term: 1 index: 461. (index mismatch)
I20240501 14:01:02.338058 24724 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 460. Preceding OpId from leader: term: 1 index: 461. (index mismatch)
I20240501 14:01:02.338791 28950 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I20240501 14:01:02.339115 28951 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I20240501 14:01:02.347270 28759 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.349275 28617 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.352237 24724 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.353755 24591 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.352026 25114 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.358412 24451 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.365689 24987 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.461: config changed from index 460 to 461, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 461 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } attrs { replace: true } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.370209 25242 consensus_queue.cc:237] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 461, Committed index: 461, Last appended: 1.461, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.373077 28617 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
I20240501 14:01:02.373095 24591 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
I20240501 14:01:02.373555 25114 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
I20240501 14:01:02.373845 28950 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 461, Time since last communication: 0.000s
I20240501 14:01:02.374279 28759 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 461, Time since last communication: 0.000s
I20240501 14:01:02.374524 24724 raft_consensus.cc:1261] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
I20240501 14:01:02.374722 28951 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 461, Time since last communication: 0.000s
I20240501 14:01:02.375192 28951 consensus_queue.cc:1035] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 461, Time since last communication: 0.000s
I20240501 14:01:02.377404 28950 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.462: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.378813 25114 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.380115 24724 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.382434 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfe9cb33f20b45aa90b066e210804355 with cas_config_opid_index 461: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:02.383924 28617 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.385324 24454 catalog_manager.cc:5484] T dfe9cb33f20b45aa90b066e210804355 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.391316 24965 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe9cb33f20b45aa90b066e210804355 with delete_type TABLET_DATA_TOMBSTONED (TS 4e03e0d09b974c7d8cf7f49652d4fd23 not found in new config with opid_index 462) from {username='slave'} at 127.0.0.1:49768
I20240501 14:01:02.393139 24591 raft_consensus.cc:2932] T dfe9cb33f20b45aa90b066e210804355 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 461 to 462, VOTER 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.398269 29010 tablet_replica.cc:330] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: stopping tablet replica
I20240501 14:01:02.398522 29010 raft_consensus.cc:2229] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.398862 29010 raft_consensus.cc:2258] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.400877 29010 ts_tablet_manager.cc:1875] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:02.407258 29010 ts_tablet_manager.cc:1888] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.461
I20240501 14:01:02.407382 29010 log.cc:1196] T dfe9cb33f20b45aa90b066e210804355 P 4e03e0d09b974c7d8cf7f49652d4fd23: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfe9cb33f20b45aa90b066e210804355
I20240501 14:01:02.408162 24439 catalog_manager.cc:4830] TS 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): tablet dfe9cb33f20b45aa90b066e210804355 (table rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]) successfully deleted
I20240501 14:01:02.424013 28617 raft_consensus.cc:1203] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.399->[1.400-1.400] Dedup: 1.400->[]
I20240501 14:01:02.426281 28617 raft_consensus.cc:1203] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.666->[1.667-1.667] Dedup: 1.667->[]
I20240501 14:01:02.520454 29042 raft_consensus.cc:1050] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:02.521551 29042 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.523118 24724 consensus_queue.cc:237] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:02.525280 26218 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240501 14:01:02.525951 24987 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240501 14:01:02.526124 25114 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.527618 28617 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240501 14:01:02.528532 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20240501 14:01:02.529322 28801 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20240501 14:01:02.533878 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20240501 14:01:02.552987 28777 raft_consensus.cc:1050] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:02.554469 28777 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 1.446, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.566773 28616 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240501 14:01:02.567584 28935 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 14:01:02.575794 24984 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240501 14:01:02.576553 28935 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 14:01:02.589463 24854 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240501 14:01:02.590296 28935 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 14:01:02.602092 26214 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240501 14:01:02.602103 24853 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.603343 25242 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.603219 28801 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.604319 26214 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.605300 29009 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:02.605610 28751 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:02.605799 25242 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240501 14:01:02.606086 28751 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:02.606859 24591 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240501 14:01:02.608111 28935 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 14:01:02.607795 26214 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.610786 28751 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:02.615072 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.009s
I20240501 14:01:02.615585 29042 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20240501 14:01:02.624362 28617 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.632400 24448 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 reported cstate change: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.639326 24984 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.658290 24724 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240501 14:01:02.659590 29055 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.660660 25116 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.662490 28777 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 14:01:02.668838 24451 catalog_manager.cc:5484] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:02.670922 25244 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
W20240501 14:01:02.675825 24663 consensus_peers.cc:485] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9419b1ee84b4b23b22a8394be3162b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:02.680958 25242 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.686178 28935 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.694723 24854 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.685586 24591 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.695631 26218 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.704562 24454 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:02.711061 24984 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.719151 26214 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.721093 28616 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.748196 24591 consensus_queue.cc:237] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 1.447, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.748586 24724 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.750949 28616 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:02.751106 26214 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:02.754542 29069 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:02.756269 24853 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.759585 29074 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:02.761698 24984 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:02.763096 29074 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [term 1 LEADER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.763661 28616 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.768301 24439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94749eeb6ba44d3ea0803cacf47017ea with cas_config_opid_index 447: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:02.769990 26214 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.773007 28777 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:02.772905 24451 catalog_manager.cc:5484] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "283529d70ba14547b9440a16350b629e" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.777356 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet 94749eeb6ba44d3ea0803cacf47017ea with delete_type TABLET_DATA_TOMBSTONED (TS 909bf3bda47b4c7ab369f460f5e96973 not found in new config with opid_index 448) from {username='slave'} at 127.0.0.1:51724
I20240501 14:01:02.782867 24984 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.786610 29076 tablet_replica.cc:330] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: stopping tablet replica
I20240501 14:01:02.786919 29076 raft_consensus.cc:2229] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.787320 29076 raft_consensus.cc:2258] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.794307 24855 raft_consensus.cc:1261] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 283529d70ba14547b9440a16350b629e: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:02.795536 29076 ts_tablet_manager.cc:1875] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:02.796033 29072 consensus_queue.cc:1035] T 94749eeb6ba44d3ea0803cacf47017ea P 283529d70ba14547b9440a16350b629e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:02.797861 24854 raft_consensus.cc:2932] T 94749eeb6ba44d3ea0803cacf47017ea P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.804070 29076 ts_tablet_manager.cc:1888] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.447
I20240501 14:01:02.804224 29076 log.cc:1196] T 94749eeb6ba44d3ea0803cacf47017ea P 909bf3bda47b4c7ab369f460f5e96973: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:01:02.805846 24441 catalog_manager.cc:4830] TS 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): tablet 94749eeb6ba44d3ea0803cacf47017ea (table rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]) successfully deleted
I20240501 14:01:02.860127 24724 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 1.442, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:02.862984 25242 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240501 14:01:02.863049 24854 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
W20240501 14:01:02.863595 24663 consensus_peers.cc:485] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6b2ff2919a045aabb3c320be4b4fcfd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:02.863972 26214 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240501 14:01:02.863910 29103 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240501 14:01:02.864230 29009 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240501 14:01:02.864411 24591 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240501 14:01:02.864735 29055 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240501 14:01:02.864988 29009 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240501 14:01:02.881145 29009 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.443: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.882117 24854 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.882863 25244 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.888823 24451 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.888907 26214 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.890197 24591 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 177 to 443, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:02.916707 28961 raft_consensus.cc:1050] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:02.917695 28961 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 1.400, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:02.920712 24854 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20240501 14:01:02.920871 26214 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20240501 14:01:02.921617 28957 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240501 14:01:02.922902 24591 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20240501 14:01:02.924619 24984 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20240501 14:01:02.925709 28616 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20240501 14:01:02.925784 28957 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240501 14:01:02.926499 28957 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240501 14:01:02.926987 29124 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240501 14:01:02.927688 29120 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240501 14:01:02.931576 28957 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.932315 28616 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.937188 24854 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.938766 24448 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.941198 26214 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.952162 24984 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.956379 24591 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:02.971043 29133 ts_tablet_manager.cc:902] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 14:01:02.980563 29133 tablet_copy_client.cc:308] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:01:02.996783 29144 raft_consensus.cc:1050] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:02.998026 29144 consensus_queue.cc:237] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 667, Committed index: 667, Last appended: 1.667, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.008283 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet d9419b1ee84b4b23b22a8394be3162b8 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:47443)
I20240501 14:01:03.008523 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet d9419b1ee84b4b23b22a8394be3162b8 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:47443: session id = c82fcb40e5534742981e8300df76a1f7-d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:01:03.011358 24747 tablet_copy_source_session.cc:205] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:03.013033 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9419b1ee84b4b23b22a8394be3162b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.015136 24854 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 1 index: 668. (index mismatch)
I20240501 14:01:03.015296 28616 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 1 index: 668. (index mismatch)
I20240501 14:01:03.016861 28903 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 667, Time since last communication: 0.000s
I20240501 14:01:03.017279 29145 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 667, Time since last communication: 0.000s
I20240501 14:01:03.019121 24724 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 1 index: 668. (index mismatch)
I20240501 14:01:03.019295 29133 tablet_copy_client.cc:783] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:03.019798 29133 tablet_copy_client.cc:647] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:03.020790 29152 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 667, Time since last communication: 0.000s
I20240501 14:01:03.021932 25116 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 1 index: 668. (index mismatch)
I20240501 14:01:03.022823 29145 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 667, Time since last communication: 0.000s
I20240501 14:01:03.029398 29133 tablet_copy_client.cc:523] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.034080 29133 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:03.037536 29145 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.041986 24725 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.046099 25116 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.052829 24454 catalog_manager.cc:5484] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.062134 24591 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 1 index: 668. (index mismatch)
I20240501 14:01:03.062525 24854 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.062750 28616 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.070022 29133 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.069962 29144 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 667, Time since last communication: 0.000s
I20240501 14:01:03.070503 29133 tablet_bootstrap.cc:492] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:03.073554 29133 ts_tablet_manager.cc:1367] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.012s
I20240501 14:01:03.074668 29133 raft_consensus.cc:348] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.075126 29133 raft_consensus.cc:729] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:03.075467 29133 consensus_queue.cc:260] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.076332 29133 ts_tablet_manager.cc:1398] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 14:01:03.077327 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-d9419b1ee84b4b23b22a8394be3162b8 received from {username='slave'} at 127.22.5.200:47443
I20240501 14:01:03.077595 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-d9419b1ee84b4b23b22a8394be3162b8 on tablet d9419b1ee84b4b23b22a8394be3162b8 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:03.081540 24591 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.668: config changed from index 667 to 668, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 668 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.228080 24854 consensus_queue.cc:237] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.230840 26218 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20240501 14:01:03.230949 28616 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20240501 14:01:03.231829 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240501 14:01:03.232461 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240501 14:01:03.233855 24591 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20240501 14:01:03.234375 24984 raft_consensus.cc:1261] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer f74cb76209d04397a8a206f86e91ac5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20240501 14:01:03.235842 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240501 14:01:03.236447 28938 consensus_queue.cc:1035] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240501 14:01:03.237089 28938 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.243758 24591 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.244747 26218 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.248703 24448 catalog_manager.cc:5484] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "f74cb76209d04397a8a206f86e91ac5b" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:03.249622 28616 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.251868 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:03.254457 24984 raft_consensus.cc:2932] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.255144 25226 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 with delete_type TABLET_DATA_TOMBSTONED (TS 84c5917a503d4a82bb2d9c0d5fdb04bd not found in new config with opid_index 321) from {username='slave'} at 127.0.0.1:45540
I20240501 14:01:03.255960 29168 tablet_replica.cc:330] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: stopping tablet replica
I20240501 14:01:03.256247 29168 raft_consensus.cc:2229] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.256539 29168 raft_consensus.cc:2258] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.258800 29168 ts_tablet_manager.cc:1875] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:03.266641 29168 ts_tablet_manager.cc:1888] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.320
I20240501 14:01:03.266880 29168 log.cc:1196] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:01:03.267652 24442 catalog_manager.cc:4830] TS 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877): tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0 (table rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]) successfully deleted
I20240501 14:01:03.309500 29133 ts_tablet_manager.cc:902] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 14:01:03.310936 29133 tablet_copy_client.cc:308] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:01:03.311722 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet c6b2ff2919a045aabb3c320be4b4fcfd from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:47443)
I20240501 14:01:03.311914 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet c6b2ff2919a045aabb3c320be4b4fcfd from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:47443: session id = c82fcb40e5534742981e8300df76a1f7-c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:01:03.314571 24747 tablet_copy_source_session.cc:205] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:03.315855 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b2ff2919a045aabb3c320be4b4fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.320981 29133 tablet_copy_client.cc:783] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:03.321233 29133 tablet_copy_client.cc:647] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:03.321506 24984 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 412, Committed index: 412, Last appended: 1.412, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.323884 29133 tablet_copy_client.cc:523] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.324873 26218 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 412. Preceding OpId from leader: term: 1 index: 413. (index mismatch)
I20240501 14:01:03.325356 24854 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 412. Preceding OpId from leader: term: 1 index: 413. (index mismatch)
I20240501 14:01:03.325524 25244 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 412. Preceding OpId from leader: term: 1 index: 413. (index mismatch)
I20240501 14:01:03.326012 24724 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 412. Preceding OpId from leader: term: 1 index: 413. (index mismatch)
I20240501 14:01:03.326624 29144 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I20240501 14:01:03.326928 29153 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I20240501 14:01:03.327081 29133 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:03.327210 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
W20240501 14:01:03.327488 24923 consensus_peers.cc:485] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7c3a66fc5634404a4ee669201d3cc85. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:03.327797 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I20240501 14:01:03.333274 29144 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.413: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.333917 24854 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.413: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.340364 24454 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.341611 24724 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.413: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.345921 26218 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.413: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.359678 25244 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.413: config changed from index 113 to 413, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.503912 28616 raft_consensus.cc:1203] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:03.510571 24984 consensus_queue.cc:237] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 668, Committed index: 668, Last appended: 1.668, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.514340 28616 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 668. Preceding OpId from leader: term: 1 index: 669. (index mismatch)
I20240501 14:01:03.514914 24854 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 668. Preceding OpId from leader: term: 1 index: 669. (index mismatch)
I20240501 14:01:03.515745 29153 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 668, Time since last communication: 0.000s
I20240501 14:01:03.516944 25116 consensus_queue.cc:237] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.517030 29144 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 668, Time since last communication: 0.000s
I20240501 14:01:03.519846 28617 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 14:01:03.520241 24855 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 14:01:03.521245 26218 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 14:01:03.522280 29124 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 14:01:03.522862 29124 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 14:01:03.523350 29124 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 14:01:03.530193 29231 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.531234 24987 raft_consensus.cc:1261] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 491e21cc623d4879b09975ede207ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 14:01:03.531127 26218 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.532012 29230 consensus_queue.cc:1035] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 14:01:03.534399 24987 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.534906 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f64ab2203c9042989dbcab79bab164a6 with cas_config_opid_index 401: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:03.536772 29153 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.669: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.537448 24448 catalog_manager.cc:5484] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "491e21cc623d4879b09975ede207ab2d" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:03.541532 24575 tablet_service.cc:1514] Processing DeleteTablet for tablet f64ab2203c9042989dbcab79bab164a6 with delete_type TABLET_DATA_TOMBSTONED (TS 283529d70ba14547b9440a16350b629e not found in new config with opid_index 402) from {username='slave'} at 127.0.0.1:44100
I20240501 14:01:03.542156 28617 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.669: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.542932 29232 tablet_replica.cc:330] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: stopping tablet replica
I20240501 14:01:03.543160 29232 raft_consensus.cc:2229] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.543462 29232 raft_consensus.cc:2258] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.545603 29232 ts_tablet_manager.cc:1875] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:03.546859 24854 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.549662 28616 raft_consensus.cc:2932] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, VOTER 283529d70ba14547b9440a16350b629e (127.22.5.193) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.551301 29232 ts_tablet_manager.cc:1888] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.401
I20240501 14:01:03.551501 29232 log.cc:1196] T f64ab2203c9042989dbcab79bab164a6 P 283529d70ba14547b9440a16350b629e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/f64ab2203c9042989dbcab79bab164a6
I20240501 14:01:03.552971 24439 catalog_manager.cc:4830] TS 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): tablet f64ab2203c9042989dbcab79bab164a6 (table rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]) successfully deleted
I20240501 14:01:03.555972 24724 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 668. Preceding OpId from leader: term: 1 index: 669. (index mismatch)
I20240501 14:01:03.557611 28903 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 668, Time since last communication: 0.000s
I20240501 14:01:03.557911 24591 raft_consensus.cc:1261] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 668. Preceding OpId from leader: term: 1 index: 669. (index mismatch)
I20240501 14:01:03.559353 29154 consensus_queue.cc:1035] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 668, Time since last communication: 0.000s
I20240501 14:01:03.562636 24439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3dfc84e5be47406f95b6246252e8d0f5 with cas_config_opid_index 668: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:03.563700 24591 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.669: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.569710 24451 catalog_manager.cc:5484] T 3dfc84e5be47406f95b6246252e8d0f5 P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.575364 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfc84e5be47406f95b6246252e8d0f5 with delete_type TABLET_DATA_TOMBSTONED (TS 491e21cc623d4879b09975ede207ab2d not found in new config with opid_index 669) from {username='slave'} at 127.0.0.1:57230
I20240501 14:01:03.561017 24724 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.669: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.582130 24855 raft_consensus.cc:2932] T 3dfc84e5be47406f95b6246252e8d0f5 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.669: config changed from index 668 to 669, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.614403 29234 tablet_replica.cc:330] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: stopping tablet replica
I20240501 14:01:03.625445 29234 raft_consensus.cc:2229] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.625775 29234 raft_consensus.cc:2258] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.627790 29234 ts_tablet_manager.cc:1875] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:03.634707 29234 ts_tablet_manager.cc:1888] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.668
I20240501 14:01:03.634949 29234 log.cc:1196] T 3dfc84e5be47406f95b6246252e8d0f5 P 491e21cc623d4879b09975ede207ab2d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:01:03.636096 24442 catalog_manager.cc:4830] TS 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): tablet 3dfc84e5be47406f95b6246252e8d0f5 (table rebalance_test_table_1 [id=014f9343c5e744958db9b1d185f88390]) successfully deleted
I20240501 14:01:03.725673 29252 ts_tablet_manager.cc:902] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227)
I20240501 14:01:03.762421 29252 tablet_copy_client.cc:308] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:43227
I20240501 14:01:03.763172 25007 tablet_copy_service.cc:140] P 4e03e0d09b974c7d8cf7f49652d4fd23: Received BeginTabletCopySession request for tablet f7c3a66fc5634404a4ee669201d3cc85 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:50315)
I20240501 14:01:03.763329 25007 tablet_copy_service.cc:161] P 4e03e0d09b974c7d8cf7f49652d4fd23: Beginning new tablet copy session on tablet f7c3a66fc5634404a4ee669201d3cc85 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:50315: session id = c82fcb40e5534742981e8300df76a1f7-f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:01:03.765417 25007 tablet_copy_source_session.cc:205] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:03.766734 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c3a66fc5634404a4ee669201d3cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.773581 29252 tablet_copy_client.cc:783] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:03.774773 29252 tablet_copy_client.cc:647] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:03.779392 29252 tablet_copy_client.cc:523] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.783776 29252 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:03.848171 29133 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=443 overwritten=0 applied=443 ignored=0} inserts{seen=1071 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.848872 29133 tablet_bootstrap.cc:492] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:03.849350 29133 ts_tablet_manager.cc:1367] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.522s user 0.447s sys 0.062s
I20240501 14:01:03.850311 29133 raft_consensus.cc:348] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.850744 29133 raft_consensus.cc:729] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:03.851033 29133 consensus_queue.cc:260] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 443, Last appended: 1.443, Last appended by leader: 443, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.852515 29133 ts_tablet_manager.cc:1398] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:03.854686 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-c6b2ff2919a045aabb3c320be4b4fcfd received from {username='slave'} at 127.22.5.200:47443
I20240501 14:01:03.854861 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-c6b2ff2919a045aabb3c320be4b4fcfd on tablet c6b2ff2919a045aabb3c320be4b4fcfd with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:03.874984 24724 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 1.447, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:03.877691 26218 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:03.877691 25244 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:03.878621 29009 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
W20240501 14:01:03.879041 24663 consensus_peers.cc:485] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0a8a92803e1459ca3338c8054b2f8b6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:03.879142 29009 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:03.879830 25116 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:03.880388 28751 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:03.887310 24984 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 14:01:03.887001 29009 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.448: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.888375 28751 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240501 14:01:03.890367 24984 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.892585 26218 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.893347 24447 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:03.893589 25116 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.897444 29271 raft_consensus.cc:1050] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:03.898442 29271 consensus_queue.cc:237] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.902065 28616 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.902354 25242 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.902716 24854 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.902947 29009 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.903504 25114 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.904168 25244 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 258 to 448, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:03.905057 28751 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.905597 29009 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.906200 29229 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.916081 26218 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:03.916608 29271 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:03.920387 28751 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.922307 24854 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.929697 25244 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.930648 25114 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.933614 26214 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.937708 24447 catalog_manager.cc:5484] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b reported cstate change: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.961015 24724 consensus_queue.cc:237] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:03.961148 28616 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } attrs { replace: true } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.964190 25244 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:03.964242 26214 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:03.964946 25116 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:03.964956 29009 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:03.965447 29009 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:03.965692 28617 raft_consensus.cc:1261] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:03.966303 29229 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:03.968304 29229 consensus_queue.cc:1035] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:03.971513 29279 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.974884 28617 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.974967 25114 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.974041 26214 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.981226 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9419b1ee84b4b23b22a8394be3162b8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:03.982543 24454 catalog_manager.cc:5484] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 reported cstate change: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:03.988142 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet d9419b1ee84b4b23b22a8394be3162b8 with delete_type TABLET_DATA_TOMBSTONED (TS f74cb76209d04397a8a206f86e91ac5b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48194
I20240501 14:01:03.991644 25244 raft_consensus.cc:2932] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f74cb76209d04397a8a206f86e91ac5b (127.22.5.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.001412 29291 tablet_replica.cc:330] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: stopping tablet replica
I20240501 14:01:04.001868 29291 raft_consensus.cc:2229] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.002298 29291 raft_consensus.cc:2258] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.003115 25244 consensus_queue.cc:237] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.004361 29291 ts_tablet_manager.cc:1875] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.006379 24984 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
W20240501 14:01:04.006978 25183 consensus_peers.cc:485] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067): Couldn't send request to peer c82fcb40e5534742981e8300df76a1f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dbc87150c2745c3854334dad7b7ed75. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:04.007038 28950 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240501 14:01:04.013667 29291 ts_tablet_manager.cc:1888] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:04.013904 29291 log.cc:1196] T d9419b1ee84b4b23b22a8394be3162b8 P f74cb76209d04397a8a206f86e91ac5b: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:01:04.014930 24442 catalog_manager.cc:4830] TS f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): tablet d9419b1ee84b4b23b22a8394be3162b8 (table TestTable [id=037bfbab72cb4186830bb25698a63714]) successfully deleted
I20240501 14:01:04.018813 24724 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240501 14:01:04.018918 24591 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240501 14:01:04.019857 28950 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240501 14:01:04.020484 28950 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240501 14:01:04.020640 25114 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240501 14:01:04.021307 28950 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240501 14:01:04.033399 28950 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.039428 25116 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.045317 24451 catalog_manager.cc:5484] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d reported cstate change: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.047250 24984 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.054142 24591 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.054703 24724 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index -1 to 316, NON_VOTER c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } } }
I20240501 14:01:04.309784 29133 ts_tablet_manager.cc:902] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 14:01:04.320765 29133 tablet_copy_client.cc:308] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:45987
I20240501 14:01:04.321724 24747 tablet_copy_service.cc:140] P 909bf3bda47b4c7ab369f460f5e96973: Received BeginTabletCopySession request for tablet c0a8a92803e1459ca3338c8054b2f8b6 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:47443)
I20240501 14:01:04.321920 24747 tablet_copy_service.cc:161] P 909bf3bda47b4c7ab369f460f5e96973: Beginning new tablet copy session on tablet c0a8a92803e1459ca3338c8054b2f8b6 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:47443: session id = c82fcb40e5534742981e8300df76a1f7-c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:01:04.324290 24747 tablet_copy_source_session.cc:205] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:04.325637 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a8a92803e1459ca3338c8054b2f8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.332017 29133 tablet_copy_client.cc:783] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:04.332298 29133 tablet_copy_client.cc:647] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:04.335223 29133 tablet_copy_client.cc:523] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:04.338564 29133 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:04.349429 29252 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=413 overwritten=0 applied=413 ignored=0} inserts{seen=1095 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:04.349887 29252 tablet_bootstrap.cc:492] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:04.350210 29252 ts_tablet_manager.cc:1367] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.567s user 0.446s sys 0.034s
I20240501 14:01:04.350916 29252 raft_consensus.cc:348] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.351152 29252 raft_consensus.cc:729] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:04.351361 29252 consensus_queue.cc:260] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 413, Last appended: 1.413, Last appended by leader: 413, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 413 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.351897 29252 ts_tablet_manager.cc:1398] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:04.352919 25007 tablet_copy_service.cc:342] P 4e03e0d09b974c7d8cf7f49652d4fd23: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-f7c3a66fc5634404a4ee669201d3cc85 received from {username='slave'} at 127.22.5.200:50315
I20240501 14:01:04.353067 25007 tablet_copy_service.cc:433] P 4e03e0d09b974c7d8cf7f49652d4fd23: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-f7c3a66fc5634404a4ee669201d3cc85 on tablet f7c3a66fc5634404a4ee669201d3cc85 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:04.385988 28617 raft_consensus.cc:1203] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.442->[1.443-1.443] Dedup: 1.443->[]
I20240501 14:01:04.536875 29252 ts_tablet_manager.cc:902] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Initiating tablet copy from peer 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 14:01:04.541814 29252 tablet_copy_client.cc:308] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:32877
I20240501 14:01:04.542546 25267 tablet_copy_service.cc:140] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Received BeginTabletCopySession request for tablet 2dbc87150c2745c3854334dad7b7ed75 from peer c82fcb40e5534742981e8300df76a1f7 ({username='slave'} at 127.22.5.200:45049)
I20240501 14:01:04.542732 25267 tablet_copy_service.cc:161] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Beginning new tablet copy session on tablet 2dbc87150c2745c3854334dad7b7ed75 from peer c82fcb40e5534742981e8300df76a1f7 at {username='slave'} at 127.22.5.200:45049: session id = c82fcb40e5534742981e8300df76a1f7-2dbc87150c2745c3854334dad7b7ed75
I20240501 14:01:04.545015 25267 tablet_copy_source_session.cc:205] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:04.546309 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbc87150c2745c3854334dad7b7ed75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:04.551226 29252 tablet_copy_client.cc:783] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:04.551518 29252 tablet_copy_client.cc:647] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:04.557305 29252 tablet_copy_client.cc:523] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:04.564568 29252 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Bootstrap starting.
I20240501 14:01:04.833369 29279 raft_consensus.cc:1050] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:04.834456 29279 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 443, Committed index: 443, Last appended: 1.443, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:04.836124 29133 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=448 overwritten=0 applied=448 ignored=0} inserts{seen=1117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:04.836750 29133 tablet_bootstrap.cc:492] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:04.837200 29133 ts_tablet_manager.cc:1367] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.499s user 0.404s sys 0.071s
I20240501 14:01:04.838153 29133 raft_consensus.cc:348] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.838579 29133 raft_consensus.cc:729] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:04.839267 29133 consensus_queue.cc:260] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 448, Last appended: 1.448, Last appended by leader: 448, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.840265 29133 ts_tablet_manager.cc:1398] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:04.841297 24747 tablet_copy_service.cc:342] P 909bf3bda47b4c7ab369f460f5e96973: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-c0a8a92803e1459ca3338c8054b2f8b6 received from {username='slave'} at 127.22.5.200:47443
I20240501 14:01:04.841454 24747 tablet_copy_service.cc:433] P 909bf3bda47b4c7ab369f460f5e96973: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-c0a8a92803e1459ca3338c8054b2f8b6 on tablet c0a8a92803e1459ca3338c8054b2f8b6 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:04.845453 24854 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240501 14:01:04.846184 28617 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240501 14:01:04.846390 26218 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240501 14:01:04.847199 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240501 14:01:04.847865 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240501 14:01:04.848436 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240501 14:01:04.849546 24591 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240501 14:01:04.850224 29279 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240501 14:01:04.858356 25244 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240501 14:01:04.859254 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240501 14:01:04.885876 25244 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.884984 29009 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.885907 26214 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.894850 24591 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.895303 28617 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.896188 24448 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.912923 28617 raft_consensus.cc:1203] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.412->[1.413-1.413] Dedup: 1.413->[]
I20240501 14:01:04.912629 24854 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false replace: true } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.922130 28617 raft_consensus.cc:1203] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.447->[1.448-1.448] Dedup: 1.448->[]
I20240501 14:01:04.930758 24725 consensus_queue.cc:237] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 1.444, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:04.933306 24854 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240501 14:01:04.933372 28617 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240501 14:01:04.933543 25244 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240501 14:01:04.933956 29398 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240501 14:01:04.934679 29009 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240501 14:01:04.935192 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240501 14:01:04.937250 24591 raft_consensus.cc:1261] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240501 14:01:04.938024 29286 consensus_queue.cc:1035] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240501 14:01:04.940661 29398 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.941681 28617 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.942078 25244 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.945111 24591 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.948196 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6b2ff2919a045aabb3c320be4b4fcfd with cas_config_opid_index 444: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.948292 24452 catalog_manager.cc:5484] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 reported cstate change: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.948828 24854 raft_consensus.cc:2932] T c6b2ff2919a045aabb3c320be4b4fcfd P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:04.970957 26198 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b2ff2919a045aabb3c320be4b4fcfd with delete_type TABLET_DATA_TOMBSTONED (TS 2f6d2177839a452eba5e2fc39cb40881 not found in new config with opid_index 445) from {username='slave'} at 127.0.0.1:53056
I20240501 14:01:04.977702 29422 tablet_replica.cc:330] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: stopping tablet replica
I20240501 14:01:04.978224 29422 raft_consensus.cc:2229] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.978762 29422 raft_consensus.cc:2258] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.984359 29422 ts_tablet_manager.cc:1875] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.994048 29422 ts_tablet_manager.cc:1888] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.444
I20240501 14:01:04.994304 29422 log.cc:1196] T c6b2ff2919a045aabb3c320be4b4fcfd P 2f6d2177839a452eba5e2fc39cb40881: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/wals/c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:01:04.995374 24442 catalog_manager.cc:4830] TS 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589): tablet c6b2ff2919a045aabb3c320be4b4fcfd (table rebalance_test_table_0 [id=9ad2a1a0f3c74e2db1f549e28d70b57f]) successfully deleted
I20240501 14:01:04.996552 29252 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=316 overwritten=0 applied=316 ignored=0} inserts{seen=988 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:04.996923 29252 tablet_bootstrap.cc:492] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Bootstrap complete.
I20240501 14:01:04.997220 29252 ts_tablet_manager.cc:1367] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Time spent bootstrapping tablet: real 0.433s user 0.348s sys 0.037s
I20240501 14:01:04.997979 29252 raft_consensus.cc:348] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.998180 29252 raft_consensus.cc:729] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82fcb40e5534742981e8300df76a1f7, State: Initialized, Role: LEARNER
I20240501 14:01:04.998411 29252 consensus_queue.cc:260] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 316, Last appended: 1.316, Last appended by leader: 316, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: true } }
I20240501 14:01:04.999336 29252 ts_tablet_manager.cc:1398] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:05.000241 25267 tablet_copy_service.cc:342] P 84c5917a503d4a82bb2d9c0d5fdb04bd: Request end of tablet copy session c82fcb40e5534742981e8300df76a1f7-2dbc87150c2745c3854334dad7b7ed75 received from {username='slave'} at 127.22.5.200:45049
I20240501 14:01:05.000355 25267 tablet_copy_service.cc:433] P 84c5917a503d4a82bb2d9c0d5fdb04bd: ending tablet copy session c82fcb40e5534742981e8300df76a1f7-2dbc87150c2745c3854334dad7b7ed75 on tablet 2dbc87150c2745c3854334dad7b7ed75 with peer c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:05.126744 28617 raft_consensus.cc:1203] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.315->[1.316-1.316] Dedup: 1.316->[]
I20240501 14:01:05.443980 29009 raft_consensus.cc:1050] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:05.444991 29009 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 448, Committed index: 448, Last appended: 1.448, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.448021 26214 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 448. Preceding OpId from leader: term: 1 index: 449. (index mismatch)
I20240501 14:01:05.448158 25244 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 448. Preceding OpId from leader: term: 1 index: 449. (index mismatch)
I20240501 14:01:05.448274 25116 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 448. Preceding OpId from leader: term: 1 index: 449. (index mismatch)
I20240501 14:01:05.448482 24984 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 448. Preceding OpId from leader: term: 1 index: 449. (index mismatch)
I20240501 14:01:05.448827 28617 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 448. Preceding OpId from leader: term: 1 index: 449. (index mismatch)
I20240501 14:01:05.449064 29009 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I20240501 14:01:05.449482 29398 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I20240501 14:01:05.449764 29286 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I20240501 14:01:05.450037 29458 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I20240501 14:01:05.450932 29279 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I20240501 14:01:05.463416 29286 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.464334 28617 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.466828 25244 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.469627 24984 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.470989 24454 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 reported cstate change: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.470878 26214 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.491578 25116 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.449: config changed from index 448 to 449, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 449 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } attrs { replace: true } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.492719 29405 raft_consensus.cc:1050] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:05.493702 29405 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 413, Committed index: 413, Last appended: 1.413, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.498360 24725 consensus_queue.cc:237] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 449, Committed index: 449, Last appended: 1.449, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.501072 26214 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 449. Preceding OpId from leader: term: 1 index: 450. (index mismatch)
I20240501 14:01:05.501811 24984 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 449. Preceding OpId from leader: term: 1 index: 450. (index mismatch)
I20240501 14:01:05.502060 28617 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 449. Preceding OpId from leader: term: 1 index: 450. (index mismatch)
I20240501 14:01:05.502490 29279 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 450, Last known committed idx: 449, Time since last communication: 0.000s
I20240501 14:01:05.502704 25114 raft_consensus.cc:1261] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 909bf3bda47b4c7ab369f460f5e96973: Log matching property violated. Preceding OpId in replica: term: 1 index: 449. Preceding OpId from leader: term: 1 index: 450. (index mismatch)
I20240501 14:01:05.503191 29279 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 450, Last known committed idx: 449, Time since last communication: 0.000s
I20240501 14:01:05.505810 29286 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 450, Last known committed idx: 449, Time since last communication: 0.000s
I20240501 14:01:05.507233 29458 consensus_queue.cc:1035] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 450, Last known committed idx: 449, Time since last communication: 0.000s
I20240501 14:01:05.509936 28617 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 413. Preceding OpId from leader: term: 1 index: 414. (index mismatch)
I20240501 14:01:05.510011 26214 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 413. Preceding OpId from leader: term: 1 index: 414. (index mismatch)
I20240501 14:01:05.510520 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 414, Last known committed idx: 413, Time since last communication: 0.000s
I20240501 14:01:05.510766 29279 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 LEADER]: Committing config change with OpId 1.450: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.512379 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 414, Last known committed idx: 413, Time since last communication: 0.000s
I20240501 14:01:05.514761 24984 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.450: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.516569 28616 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.450: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.517896 24450 catalog_manager.cc:5484] T c0a8a92803e1459ca3338c8054b2f8b6 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "909bf3bda47b4c7ab369f460f5e96973" committed_config { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:05.519541 24441 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0a8a92803e1459ca3338c8054b2f8b6 with cas_config_opid_index 449: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:05.519104 25116 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.450: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.520311 26218 raft_consensus.cc:2932] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.450: config changed from index 449 to 450, VOTER 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198) evicted. New config: { opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.524468 25244 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 413. Preceding OpId from leader: term: 1 index: 414. (index mismatch)
I20240501 14:01:05.525158 24725 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 413. Preceding OpId from leader: term: 1 index: 414. (index mismatch)
I20240501 14:01:05.525959 24854 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 413. Preceding OpId from leader: term: 1 index: 414. (index mismatch)
I20240501 14:01:05.528246 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 414, Last known committed idx: 413, Time since last communication: 0.000s
I20240501 14:01:05.528813 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 414, Last known committed idx: 413, Time since last communication: 0.000s
I20240501 14:01:05.529326 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 414, Last known committed idx: 413, Time since last communication: 0.000s
I20240501 14:01:05.534207 25226 tablet_service.cc:1514] Processing DeleteTablet for tablet c0a8a92803e1459ca3338c8054b2f8b6 with delete_type TABLET_DATA_TOMBSTONED (TS 84c5917a503d4a82bb2d9c0d5fdb04bd not found in new config with opid_index 450) from {username='slave'} at 127.0.0.1:45540
I20240501 14:01:05.554279 29479 tablet_replica.cc:330] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: stopping tablet replica
I20240501 14:01:05.560161 29405 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.560854 28616 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.561498 29479 raft_consensus.cc:2229] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.561952 29479 raft_consensus.cc:2258] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.566290 26214 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.567467 25244 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.568222 24450 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 reported cstate change: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.578595 24725 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.587339 24984 consensus_queue.cc:237] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 414, Committed index: 414, Last appended: 1.414, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.589979 26214 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 414. Preceding OpId from leader: term: 1 index: 415. (index mismatch)
I20240501 14:01:05.590133 28617 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 414. Preceding OpId from leader: term: 1 index: 415. (index mismatch)
I20240501 14:01:05.591138 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I20240501 14:01:05.591516 25244 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 414. Preceding OpId from leader: term: 1 index: 415. (index mismatch)
I20240501 14:01:05.591588 24854 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.414: config changed from index 413 to 414, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 414 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } attrs { replace: true } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.592105 29473 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I20240501 14:01:05.592708 29473 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I20240501 14:01:05.596814 29154 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 LEADER]: Committing config change with OpId 1.415: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.597610 26214 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Committing config change with OpId 1.415: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.598150 25244 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Committing config change with OpId 1.415: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.596089 29479 ts_tablet_manager.cc:1875] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:05.601101 24439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7c3a66fc5634404a4ee669201d3cc85 with cas_config_opid_index 414: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:05.602934 28617 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.415: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.604741 24450 catalog_manager.cc:5484] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 reported cstate change: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" committed_config { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:05.607632 24855 raft_consensus.cc:1261] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Refusing update from remote peer 4e03e0d09b974c7d8cf7f49652d4fd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 414. Preceding OpId from leader: term: 1 index: 415. (index mismatch)
I20240501 14:01:05.609339 29154 consensus_queue.cc:1035] T f7c3a66fc5634404a4ee669201d3cc85 P 4e03e0d09b974c7d8cf7f49652d4fd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.001s
I20240501 14:01:05.610018 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c3a66fc5634404a4ee669201d3cc85 with delete_type TABLET_DATA_TOMBSTONED (TS 909bf3bda47b4c7ab369f460f5e96973 not found in new config with opid_index 415) from {username='slave'} at 127.0.0.1:51724
I20240501 14:01:05.611935 24855 raft_consensus.cc:2932] T f7c3a66fc5634404a4ee669201d3cc85 P f74cb76209d04397a8a206f86e91ac5b [term 1 FOLLOWER]: Committing config change with OpId 1.415: config changed from index 414 to 415, VOTER 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) evicted. New config: { opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.613575 29488 tablet_replica.cc:330] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: stopping tablet replica
I20240501 14:01:05.613830 29488 raft_consensus.cc:2229] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.614254 29488 raft_consensus.cc:2258] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.616604 29488 ts_tablet_manager.cc:1875] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:05.621073 29479 ts_tablet_manager.cc:1888] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.449
I20240501 14:01:05.621279 29479 log.cc:1196] T c0a8a92803e1459ca3338c8054b2f8b6 P 84c5917a503d4a82bb2d9c0d5fdb04bd: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:01:05.622787 24442 catalog_manager.cc:4830] TS 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877): tablet c0a8a92803e1459ca3338c8054b2f8b6 (table rebalance_test_table_4 [id=a722443079e943e28774847bc8347ec3]) successfully deleted
I20240501 14:01:05.623690 29488 ts_tablet_manager.cc:1888] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.414
I20240501 14:01:05.623908 29488 log.cc:1196] T f7c3a66fc5634404a4ee669201d3cc85 P 909bf3bda47b4c7ab369f460f5e96973: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:01:05.624936 24441 catalog_manager.cc:4830] TS 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): tablet f7c3a66fc5634404a4ee669201d3cc85 (table rebalance_test_table_3 [id=e9a223c9990f4a2abf67a7ef7e640f99]) successfully deleted
I20240501 14:01:05.647359 29434 raft_consensus.cc:1050] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd: attempting to promote NON_VOTER c82fcb40e5534742981e8300df76a1f7 to VOTER
I20240501 14:01:05.648211 29434 consensus_queue.cc:237] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 1.316, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.650658 28617 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 LEARNER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240501 14:01:05.651374 24984 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240501 14:01:05.651603 24725 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240501 14:01:05.651942 25116 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240501 14:01:05.652153 24591 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20240501 14:01:05.652256 29494 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240501 14:01:05.652578 29497 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240501 14:01:05.652928 29434 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240501 14:01:05.653285 29498 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240501 14:01:05.653777 29496 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240501 14:01:05.658030 29494 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.658615 24725 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.660249 25116 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.663175 24452 catalog_manager.cc:5484] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 reported cstate change: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.665099 24984 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.666261 24591 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.680738 25244 consensus_queue.cc:237] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 1.317, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:05.681227 28617 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, c82fcb40e5534742981e8300df76a1f7 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } attrs { replace: true } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.683347 24984 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20240501 14:01:05.684186 29497 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20240501 14:01:05.685329 28616 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20240501 14:01:05.685319 24725 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20240501 14:01:05.685982 29494 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20240501 14:01:05.686209 29497 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20240501 14:01:05.686537 24591 raft_consensus.cc:1261] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Refusing update from remote peer 84c5917a503d4a82bb2d9c0d5fdb04bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20240501 14:01:05.687805 29496 consensus_queue.cc:1035] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20240501 14:01:05.690157 29498 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 LEADER]: Committing config change with OpId 1.318: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.690732 24595 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.694176 24442 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dbc87150c2745c3854334dad7b7ed75 with cas_config_opid_index 317: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:05.695915 24984 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 4e03e0d09b974c7d8cf7f49652d4fd23 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.697064 24450 catalog_manager.cc:5484] T 2dbc87150c2745c3854334dad7b7ed75 P 283529d70ba14547b9440a16350b629e reported cstate change: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.697611 28616 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.702610 24725 raft_consensus.cc:2932] T 2dbc87150c2745c3854334dad7b7ed75 P 909bf3bda47b4c7ab369f460f5e96973 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, VOTER 491e21cc623d4879b09975ede207ab2d (127.22.5.197) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } } }
I20240501 14:01:05.704612 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dbc87150c2745c3854334dad7b7ed75 with delete_type TABLET_DATA_TOMBSTONED (TS 491e21cc623d4879b09975ede207ab2d not found in new config with opid_index 318) from {username='slave'} at 127.0.0.1:57230
I20240501 14:01:05.705474 29504 tablet_replica.cc:330] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: stopping tablet replica
I20240501 14:01:05.705725 29504 raft_consensus.cc:2229] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:05.706043 29504 raft_consensus.cc:2258] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:05.708264 29504 ts_tablet_manager.cc:1875] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:05.714211 29504 ts_tablet_manager.cc:1888] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.317
I20240501 14:01:05.714334 29504 log.cc:1196] T 2dbc87150c2745c3854334dad7b7ed75 P 491e21cc623d4879b09975ede207ab2d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/2dbc87150c2745c3854334dad7b7ed75
I20240501 14:01:05.715107 24442 catalog_manager.cc:4830] TS 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): tablet 2dbc87150c2745c3854334dad7b7ed75 (table rebalance_test_table_2 [id=f42e7adb897e4d9b9877c7894b6c028e]) successfully deleted
I20240501 14:01:05.965335 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:05.979504 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:05.998208 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:05.998343 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:06.018707 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.024484 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.036360 28597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.050956 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:06.261520 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.266685 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.281526 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.284386 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.285687 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:06.289558 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.317330 28597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.344075 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:06.694265 24575 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.698885 26198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.709410 24965 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.724489 25095 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.738262 24705 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:06.774140 25226 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:06.791538 28597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.798002 24836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 14:01:06.803534 25183 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.200:37067, user_credentials={real_user=slave}} blocked reactor thread for 59422.9us
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
5c4f17b3dc294f979e57abb381a3996f | 127.22.5.254:40379 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:34597 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
283529d70ba14547b9440a16350b629e | 127.22.5.193:35547 | HEALTHY | <none> | 2 | 0
2f6d2177839a452eba5e2fc39cb40881 | 127.22.5.199:35589 | HEALTHY | <none> | 0 | 0
491e21cc623d4879b09975ede207ab2d | 127.22.5.197:35069 | HEALTHY | <none> | 2 | 0
4e03e0d09b974c7d8cf7f49652d4fd23 | 127.22.5.196:43227 | HEALTHY | <none> | 2 | 0
84c5917a503d4a82bb2d9c0d5fdb04bd | 127.22.5.198:32877 | HEALTHY | <none> | 2 | 0
909bf3bda47b4c7ab369f460f5e96973 | 127.22.5.194:45987 | HEALTHY | <none> | 5 | 0
c82fcb40e5534742981e8300df76a1f7 | 127.22.5.200:37067 | HEALTHY | <none> | 0 | 0
f74cb76209d04397a8a206f86e91ac5b | 127.22.5.195:37091 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.5.193 | experimental | 127.22.5.193:35547
local_ip_for_outbound_sockets | 127.22.5.194 | experimental | 127.22.5.194:45987
local_ip_for_outbound_sockets | 127.22.5.195 | experimental | 127.22.5.195:37091
local_ip_for_outbound_sockets | 127.22.5.196 | experimental | 127.22.5.196:43227
local_ip_for_outbound_sockets | 127.22.5.197 | experimental | 127.22.5.197:35069
local_ip_for_outbound_sockets | 127.22.5.198 | experimental | 127.22.5.198:32877
local_ip_for_outbound_sockets | 127.22.5.199 | experimental | 127.22.5.199:35589
local_ip_for_outbound_sockets | 127.22.5.200 | experimental | 127.22.5.200:37067
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.5.193:35547
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.5.194:45987
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.5.195:37091
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.5.196:43227
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.5.197:35069
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.5.198:32877
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.5.199:35589
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.5.200:37067
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:34597 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:01:06.899478 22551 log_verifier.cc:126] Checking tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:01:07.210124 22551 log_verifier.cc:177] Verified matching terms for 321 ops in tablet 11f7c6d19fc7402ebc9fa3a3f07f76e0
I20240501 14:01:07.210896 22551 log_verifier.cc:126] Checking tablet 2dbc87150c2745c3854334dad7b7ed75
I20240501 14:01:07.532259 22551 log_verifier.cc:177] Verified matching terms for 318 ops in tablet 2dbc87150c2745c3854334dad7b7ed75
I20240501 14:01:07.533283 22551 log_verifier.cc:126] Checking tablet 2f1344fe8e4742e5914f49ffcaba21f8
I20240501 14:01:07.909742 22551 log_verifier.cc:177] Verified matching terms for 417 ops in tablet 2f1344fe8e4742e5914f49ffcaba21f8
I20240501 14:01:07.910689 22551 log_verifier.cc:126] Checking tablet 3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:01:08.496362 22551 log_verifier.cc:177] Verified matching terms for 669 ops in tablet 3dfc84e5be47406f95b6246252e8d0f5
I20240501 14:01:08.497905 22551 log_verifier.cc:126] Checking tablet 94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:01:08.926352 22551 log_verifier.cc:177] Verified matching terms for 448 ops in tablet 94749eeb6ba44d3ea0803cacf47017ea
I20240501 14:01:08.927423 22551 log_verifier.cc:126] Checking tablet 96a1c9e9695f4e0cbab9d05138eef65a
I20240501 14:01:09.491537 22551 log_verifier.cc:177] Verified matching terms for 649 ops in tablet 96a1c9e9695f4e0cbab9d05138eef65a
I20240501 14:01:09.492940 22551 log_verifier.cc:126] Checking tablet 9762584f4ce94ab68a52bc1f446e569c
I20240501 14:01:10.027662 22551 log_verifier.cc:177] Verified matching terms for 620 ops in tablet 9762584f4ce94ab68a52bc1f446e569c
I20240501 14:01:10.029075 22551 log_verifier.cc:126] Checking tablet 9ed1065df4344515b0f598f912e22149
I20240501 14:01:10.347779 22551 log_verifier.cc:177] Verified matching terms for 362 ops in tablet 9ed1065df4344515b0f598f912e22149
I20240501 14:01:10.348655 22551 log_verifier.cc:126] Checking tablet bb4b73ec3b4146fda1248c7cb800afbd
I20240501 14:01:10.812364 22551 log_verifier.cc:177] Verified matching terms for 467 ops in tablet bb4b73ec3b4146fda1248c7cb800afbd
I20240501 14:01:10.813655 22551 log_verifier.cc:126] Checking tablet c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:01:11.194756 22551 log_verifier.cc:177] Verified matching terms for 450 ops in tablet c0a8a92803e1459ca3338c8054b2f8b6
I20240501 14:01:11.196117 22551 log_verifier.cc:126] Checking tablet c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:01:11.591727 22551 log_verifier.cc:177] Verified matching terms for 445 ops in tablet c6b2ff2919a045aabb3c320be4b4fcfd
I20240501 14:01:11.592754 22551 log_verifier.cc:126] Checking tablet d02fe94840e24205aa98b3190d089784
I20240501 14:01:12.005771 22551 log_verifier.cc:177] Verified matching terms for 435 ops in tablet d02fe94840e24205aa98b3190d089784
I20240501 14:01:12.006830 22551 log_verifier.cc:126] Checking tablet d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:01:12.038277 22551 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d9419b1ee84b4b23b22a8394be3162b8
I20240501 14:01:12.038408 22551 log_verifier.cc:126] Checking tablet dfe9cb33f20b45aa90b066e210804355
I20240501 14:01:12.425735 22551 log_verifier.cc:177] Verified matching terms for 462 ops in tablet dfe9cb33f20b45aa90b066e210804355
I20240501 14:01:12.426775 22551 log_verifier.cc:126] Checking tablet f64ab2203c9042989dbcab79bab164a6
I20240501 14:01:12.793226 22551 log_verifier.cc:177] Verified matching terms for 402 ops in tablet f64ab2203c9042989dbcab79bab164a6
I20240501 14:01:12.794171 22551 log_verifier.cc:126] Checking tablet f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:01:13.159888 22551 log_verifier.cc:177] Verified matching terms for 415 ops in tablet f7c3a66fc5634404a4ee669201d3cc85
I20240501 14:01:13.166359 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24514
I20240501 14:01:13.173470 24636 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:13.460915 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24514
W20240501 14:01:13.511083 26146 connection.cc:535] server connection from 127.22.5.193:56803 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:13.511523 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24645
W20240501 14:01:13.511523 24920 connection.cc:535] client connection to 127.22.5.193:35547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.511593 25182 connection.cc:535] server connection from 127.22.5.193:55259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.511808 24920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.512079 25050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.5.193:35547 (error 108)
W20240501 14:01:13.512795 24662 connection.cc:535] server connection from 127.22.5.193:45021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.512912 28548 connection.cc:535] server connection from 127.22.5.193:42877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.514441 24920 consensus_peers.cc:485] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.515103 25050 consensus_peers.cc:485] T 96a1c9e9695f4e0cbab9d05138eef65a P 491e21cc623d4879b09975ede207ab2d -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.519584 24767 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:01:13.577178 24790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:13.590234 24790 consensus_peers.cc:485] T d02fe94840e24205aa98b3190d089784 P f74cb76209d04397a8a206f86e91ac5b -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.590323 25180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:13.591885 25180 consensus_peers.cc:485] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.621496 25180 consensus_peers.cc:485] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.747591 24790 consensus_peers.cc:485] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P f74cb76209d04397a8a206f86e91ac5b -> Peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Couldn't send request to peer 283529d70ba14547b9440a16350b629e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.820994 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24645
W20240501 14:01:13.869518 25182 connection.cc:535] client connection to 127.22.5.194:45987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.869561 24922 connection.cc:535] client connection to 127.22.5.194:45987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:13.870621 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24775
W20240501 14:01:13.870978 25182 consensus_peers.cc:485] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Couldn't send request to peer 909bf3bda47b4c7ab369f460f5e96973. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.871079 25182 consensus_peers.cc:485] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Couldn't send request to peer 909bf3bda47b4c7ab369f460f5e96973. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.872455 24922 consensus_peers.cc:485] T 3dfc84e5be47406f95b6246252e8d0f5 P 4e03e0d09b974c7d8cf7f49652d4fd23 -> Peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Couldn't send request to peer 909bf3bda47b4c7ab369f460f5e96973. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.877365 24897 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:14.157793 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24775
I20240501 14:01:14.200942 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24905
I20240501 14:01:14.206174 25027 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:01:14.409333 25053 consensus_peers.cc:485] T f64ab2203c9042989dbcab79bab164a6 P 491e21cc623d4879b09975ede207ab2d -> Peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Couldn't send request to peer f74cb76209d04397a8a206f86e91ac5b. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.440083 29662 raft_consensus.cc:482] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.440307 29662 raft_consensus.cc:504] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 14:01:14.441424 29662 leader_election.cc:288] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
I20240501 14:01:14.442360 25244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 417 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
W20240501 14:01:14.443127 25050 leader_election.cc:334] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:14.444294 25052 leader_election.cc:334] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.444391 25052 leader_election.cc:302] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 491e21cc623d4879b09975ede207ab2d; no voters: 283529d70ba14547b9440a16350b629e, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973
I20240501 14:01:14.444653 29662 raft_consensus.cc:2726] T 2f1344fe8e4742e5914f49ffcaba21f8 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.484339 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1344fe8e4742e5914f49ffcaba21f8" candidate_uuid: "491e21cc623d4879b09975ede207ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 417 } } ignore_live_leader: false dest_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" is_pre_election: true
I20240501 14:01:14.494766 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24905
W20240501 14:01:14.543895 24441 connection.cc:535] server connection from 127.22.5.196:38363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:14.544014 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 25035
I20240501 14:01:14.549804 25156 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:14.582742 29686 raft_consensus.cc:482] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.583098 29686 raft_consensus.cc:504] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:01:14.585712 29686 leader_election.cc:288] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877)
W20240501 14:01:14.586681 26148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
W20240501 14:01:14.588495 26148 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
W20240501 14:01:14.589577 26149 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
I20240501 14:01:14.593510 25244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 362 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 14:01:14.594087 26149 leader_election.cc:302] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881; no voters: 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:14.594414 29686 raft_consensus.cc:2726] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.631862 29686 raft_consensus.cc:482] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.632063 29686 raft_consensus.cc:504] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.633222 29686 leader_election.cc:288] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
W20240501 14:01:14.634902 26146 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
W20240501 14:01:14.635272 26148 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.640285 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 450 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:14.640938 26149 leader_election.cc:302] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881; no voters: 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973, c82fcb40e5534742981e8300df76a1f7
I20240501 14:01:14.641197 29686 raft_consensus.cc:2726] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.706485 29693 raft_consensus.cc:482] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.706799 29693 raft_consensus.cc:504] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 450 OBSOLETE_local: false peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.709103 29693 leader_election.cc:288] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 14:01:14.716449 29686 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.716629 29686 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.717590 29686 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:14.718371 25244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
I20240501 14:01:14.718746 29693 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.718943 29693 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.719455 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 450 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 14:01:14.719790 26214 raft_consensus.cc:2454] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82fcb40e5534742981e8300df76a1f7 in term 1.
I20240501 14:01:14.719866 29693 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987)
I20240501 14:01:14.720351 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 14:01:14.720511 26214 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82fcb40e5534742981e8300df76a1f7 in term 1.
I20240501 14:01:14.720867 25244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
W20240501 14:01:14.721020 28547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
W20240501 14:01:14.721431 26148 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.722460 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:14.722728 28616 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6d2177839a452eba5e2fc39cb40881 in term 1.
W20240501 14:01:14.723393 28549 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
W20240501 14:01:14.723465 28547 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
W20240501 14:01:14.723583 28549 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.781350 29671 raft_consensus.cc:482] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.781587 29671 raft_consensus.cc:504] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:01:14.782852 29671 leader_election.cc:288] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
W20240501 14:01:14.784739 25182 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
W20240501 14:01:14.784915 25183 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
I20240501 14:01:14.789620 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 362 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 14:01:14.789858 26214 raft_consensus.cc:2454] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 14:01:14.808627 29671 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.808880 29671 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.809880 29671 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:14.810390 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 14:01:14.810447 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:14.810586 26214 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 14:01:14.810686 28616 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 14:01:14.811233 25183 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, 84c5917a503d4a82bb2d9c0d5fdb04bd, c82fcb40e5534742981e8300df76a1f7; no voters:
W20240501 14:01:14.811450 25182 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.811534 29671 raft_consensus.cc:2781] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.811659 29671 raft_consensus.cc:482] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.811748 29671 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.813647 29671 raft_consensus.cc:504] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.814435 29671 leader_election.cc:288] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 election: Requested vote from peers 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:14.814630 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881"
I20240501 14:01:14.814857 26214 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.814862 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7"
I20240501 14:01:14.815052 28616 raft_consensus.cc:3037] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:14.816032 25182 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.817631 28616 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 2.
I20240501 14:01:14.817880 26214 raft_consensus.cc:2454] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 2.
I20240501 14:01:14.818261 25183 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, 84c5917a503d4a82bb2d9c0d5fdb04bd, c82fcb40e5534742981e8300df76a1f7; no voters: 909bf3bda47b4c7ab369f460f5e96973
I20240501 14:01:14.818509 29671 raft_consensus.cc:2781] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.818666 29671 raft_consensus.cc:686] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 2 LEADER]: Becoming Leader. State: Replica: 84c5917a503d4a82bb2d9c0d5fdb04bd, State: Running, Role: LEADER
I20240501 14:01:14.818959 29671 consensus_queue.cc:237] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.822417 24451 catalog_manager.cc:5484] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd reported cstate change: term changed from 1 to 2, leader changed from 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194) to 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198). New cstate: current_term: 2 leader_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.830201 29693 raft_consensus.cc:482] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.830396 29693 raft_consensus.cc:504] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
W20240501 14:01:14.830600 25180 consensus_peers.cc:485] T 2dbc87150c2745c3854334dad7b7ed75 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Couldn't send request to peer 4e03e0d09b974c7d8cf7f49652d4fd23. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.831501 29693 leader_election.cc:288] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091)
I20240501 14:01:14.831707 25244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 445 } } ignore_live_leader: false dest_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" is_pre_election: true
W20240501 14:01:14.833194 28547 leader_election.cc:334] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:14.833835 28550 leader_election.cc:334] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
I20240501 14:01:14.833994 28550 leader_election.cc:302] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c82fcb40e5534742981e8300df76a1f7; no voters: 283529d70ba14547b9440a16350b629e, 84c5917a503d4a82bb2d9c0d5fdb04bd, f74cb76209d04397a8a206f86e91ac5b
W20240501 14:01:14.834225 28549 leader_election.cc:334] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987): Network error: Client connection negotiation failed: client connection to 127.22.5.194:45987: connect: Connection refused (error 111)
I20240501 14:01:14.834349 29693 raft_consensus.cc:2726] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.848409 29706 raft_consensus.cc:482] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:14.848451 29704 raft_consensus.cc:482] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 283529d70ba14547b9440a16350b629e)
I20240501 14:01:14.848716 29706 raft_consensus.cc:504] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:01:14.848778 29704 raft_consensus.cc:504] T bb4b73ec3b4146fda1248c7cb800afbd P 491e21cc623d4879b09975ede207ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 14:01:14.849081 25114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a8a92803e1459ca3338c8054b2f8b6" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 450 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 14:01:14.849584 25116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9419b1ee84b4b23b22a8394be3162b8" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
I20240501 14:01:14.849792 29706 leader_election.cc:288] T 9ed1065df4344515b0f598f912e22149 P 491e21cc623d4879b09975ede207ab2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 84c5917a503d4a82bb2d9c0d5fdb04bd (127.22.5.198:32877), 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 14:01:14.849992 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 25035
I20240501 14:01:14.849978 25117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed1065df4344515b0f598f912e22149" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 362 } } ignore_live_leader: false dest_uuid: "491e21cc623d4879b09975ede207ab2d" is_pre_election: true
W20240501 14:01:14.899626 25183 connection.cc:535] client connection to 127.22.5.197:35069 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:14.900308 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 25165
W20240501 14:01:14.901654 25183 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.901669 26149 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.901722 28550 leader_election.cc:334] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
I20240501 14:01:14.901786 25183 leader_election.cc:302] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, 84c5917a503d4a82bb2d9c0d5fdb04bd; no voters: 491e21cc623d4879b09975ede207ab2d, 909bf3bda47b4c7ab369f460f5e96973, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:14.901854 28550 leader_election.cc:302] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, c82fcb40e5534742981e8300df76a1f7; no voters: 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, 909bf3bda47b4c7ab369f460f5e96973
I20240501 14:01:14.902091 29671 raft_consensus.cc:2726] T 9ed1065df4344515b0f598f912e22149 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:14.902215 26149 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.902243 28550 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.902251 25183 consensus_peers.cc:485] T dfe9cb33f20b45aa90b066e210804355 P 84c5917a503d4a82bb2d9c0d5fdb04bd -> Peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Couldn't send request to peer 491e21cc623d4879b09975ede207ab2d. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.902328 26149 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, c82fcb40e5534742981e8300df76a1f7; no voters: 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973
I20240501 14:01:14.902350 28550 leader_election.cc:302] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, c82fcb40e5534742981e8300df76a1f7; no voters: 491e21cc623d4879b09975ede207ab2d, 84c5917a503d4a82bb2d9c0d5fdb04bd, 909bf3bda47b4c7ab369f460f5e96973
W20240501 14:01:14.902367 25183 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
I20240501 14:01:14.902431 29693 raft_consensus.cc:2726] T c0a8a92803e1459ca3338c8054b2f8b6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.902593 29693 raft_consensus.cc:2726] T d9419b1ee84b4b23b22a8394be3162b8 P c82fcb40e5534742981e8300df76a1f7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:14.902644 25183 leader_election.cc:334] T d9419b1ee84b4b23b22a8394be3162b8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.902738 26149 leader_election.cc:334] T 9ed1065df4344515b0f598f912e22149 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
I20240501 14:01:14.903095 29686 raft_consensus.cc:2726] T d9419b1ee84b4b23b22a8394be3162b8 P 2f6d2177839a452eba5e2fc39cb40881 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.907578 25286 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:14.931529 29686 raft_consensus.cc:482] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f74cb76209d04397a8a206f86e91ac5b)
I20240501 14:01:14.931752 29686 raft_consensus.cc:504] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } }
I20240501 14:01:14.932744 29686 leader_election.cc:288] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
W20240501 14:01:14.934770 26149 leader_election.cc:334] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
W20240501 14:01:14.934919 26146 leader_election.cc:334] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:14.935000 26149 leader_election.cc:334] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069): Network error: Client connection negotiation failed: client connection to 127.22.5.197:35069: connect: Connection refused (error 111)
W20240501 14:01:14.935081 26146 leader_election.cc:334] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
I20240501 14:01:14.935145 26149 leader_election.cc:302] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 1 yes votes; 4 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881; no voters: 283529d70ba14547b9440a16350b629e, 491e21cc623d4879b09975ede207ab2d, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:14.935372 29686 raft_consensus.cc:2726] T d02fe94840e24205aa98b3190d089784 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.962788 29686 raft_consensus.cc:482] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 283529d70ba14547b9440a16350b629e)
I20240501 14:01:14.962719 29708 raft_consensus.cc:482] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 283529d70ba14547b9440a16350b629e)
I20240501 14:01:14.962970 29686 raft_consensus.cc:504] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.962985 29708 raft_consensus.cc:504] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.964043 29686 leader_election.cc:288] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:14.964150 29708 leader_election.cc:288] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 14:01:14.965350 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 448 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:14.965633 28616 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6d2177839a452eba5e2fc39cb40881 in term 1.
I20240501 14:01:14.966307 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94749eeb6ba44d3ea0803cacf47017ea" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 448 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
I20240501 14:01:14.966514 26214 raft_consensus.cc:2454] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82fcb40e5534742981e8300df76a1f7 in term 1.
W20240501 14:01:14.967521 26146 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:14.967764 28550 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
W20240501 14:01:14.967939 26146 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
W20240501 14:01:14.968071 26149 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
I20240501 14:01:14.968142 26149 leader_election.cc:302] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881; no voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b
W20240501 14:01:14.968428 28547 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
I20240501 14:01:14.968469 29686 raft_consensus.cc:2726] T 94749eeb6ba44d3ea0803cacf47017ea P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:14.968650 28547 leader_election.cc:334] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
I20240501 14:01:14.968729 28547 leader_election.cc:302] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, c82fcb40e5534742981e8300df76a1f7; no voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:14.968966 29708 raft_consensus.cc:2726] T 94749eeb6ba44d3ea0803cacf47017ea P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.992777 29708 raft_consensus.cc:482] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f74cb76209d04397a8a206f86e91ac5b)
I20240501 14:01:14.993011 29708 raft_consensus.cc:504] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:14.994060 29708 leader_election.cc:288] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), 2f6d2177839a452eba5e2fc39cb40881 (127.22.5.199:35589)
I20240501 14:01:14.994836 26214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "c82fcb40e5534742981e8300df76a1f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 321 } } ignore_live_leader: false dest_uuid: "2f6d2177839a452eba5e2fc39cb40881" is_pre_election: true
W20240501 14:01:14.995755 28550 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
W20240501 14:01:14.996013 28547 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
I20240501 14:01:14.996145 28547 leader_election.cc:302] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c82fcb40e5534742981e8300df76a1f7; no voters: 283529d70ba14547b9440a16350b629e, 2f6d2177839a452eba5e2fc39cb40881, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:14.996398 29708 raft_consensus.cc:2726] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:14.996816 28547 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
I20240501 14:01:15.038022 29686 raft_consensus.cc:482] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f74cb76209d04397a8a206f86e91ac5b)
I20240501 14:01:15.038235 29686 raft_consensus.cc:504] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:15.039172 29686 leader_election.cc:288] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:15.040361 28616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f7c6d19fc7402ebc9fa3a3f07f76e0" candidate_uuid: "2f6d2177839a452eba5e2fc39cb40881" candidate_term: 2 candidate_status { last_received { term: 1 index: 321 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:15.040648 28616 raft_consensus.cc:2454] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6d2177839a452eba5e2fc39cb40881 in term 1.
W20240501 14:01:15.040948 26149 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091): Network error: Client connection negotiation failed: client connection to 127.22.5.195:37091: connect: Connection refused (error 111)
W20240501 14:01:15.041172 26146 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 283529d70ba14547b9440a16350b629e (127.22.5.193:35547): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35547: connect: Connection refused (error 111)
W20240501 14:01:15.041365 26146 leader_election.cc:334] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e03e0d09b974c7d8cf7f49652d4fd23 (127.22.5.196:43227): Network error: Client connection negotiation failed: client connection to 127.22.5.196:43227: connect: Connection refused (error 111)
I20240501 14:01:15.041465 26146 leader_election.cc:302] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2f6d2177839a452eba5e2fc39cb40881, c82fcb40e5534742981e8300df76a1f7; no voters: 283529d70ba14547b9440a16350b629e, 4e03e0d09b974c7d8cf7f49652d4fd23, f74cb76209d04397a8a206f86e91ac5b
I20240501 14:01:15.041702 29686 raft_consensus.cc:2726] T 11f7c6d19fc7402ebc9fa3a3f07f76e0 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:15.179394 29671 raft_consensus.cc:482] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 283529d70ba14547b9440a16350b629e)
I20240501 14:01:15.180042 29671 raft_consensus.cc:504] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } }
I20240501 14:01:15.180227 29711 raft_consensus.cc:482] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:15.180366 29672 raft_consensus.cc:482] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 909bf3bda47b4c7ab369f460f5e96973)
I20240501 14:01:15.180552 29672 raft_consensus.cc:504] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:15.181138 29671 leader_election.cc:288] T bb4b73ec3b4146fda1248c7cb800afbd P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), 491e21cc623d4879b09975ede207ab2d (127.22.5.197:35069)
I20240501 14:01:15.181361 29672 leader_election.cc:288] T c6b2ff2919a045aabb3c320be4b4fcfd P 84c5917a503d4a82bb2d9c0d5fdb04bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 283529d70ba14547b9440a16350b629e (127.22.5.193:35547), 909bf3bda47b4c7ab369f460f5e96973 (127.22.5.194:45987), f74cb76209d04397a8a206f86e91ac5b (127.22.5.195:37091), c82fcb40e5534742981e8300df76a1f7 (127.22.5.200:37067)
I20240501 14:01:15.181936 28617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b2ff2919a045aabb3c320be4b4fcfd" candidate_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 445 } } ignore_live_leader: false dest_uuid: "c82fcb40e5534742981e8300df76a1f7" is_pre_election: true
I20240501 14:01:15.182211 28617 raft_consensus.cc:2454] T c6b2ff2919a045aabb3c320be4b4fcfd P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c5917a503d4a82bb2d9c0d5fdb04bd in term 1.
I20240501 14:01:15.181679 29711 raft_consensus.cc:504] T 2f1344fe8e4742e5914f49ffcaba21f8 P 84c5917a503d4a82bb2d9c0d5fdb04bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283529d70ba14547b9440a16350b629e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35547 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "909bf3bda47b4c7ab369f460f5e96973" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45987 } } peers { permanent_uuid: "491e21cc623d4879b09975ede207ab2d" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 35069 } } peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } }
I20240501 14:01:15.183315 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 25165
W20240501 14:01:15.232461 28548 connection.cc:535] server connection from 127.22.5.198:42639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:15.232832 24442 connection.cc:535] server connection from 127.22.5.198:57533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:15.232951 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 26131
I20240501 14:01:15.238477 26257 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:15.484050 29686 raft_consensus.cc:482] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e03e0d09b974c7d8cf7f49652d4fd23)
I20240501 14:01:15.484364 29686 raft_consensus.cc:504] T f7c3a66fc5634404a4ee669201d3cc85 P 2f6d2177839a452eba5e2fc39cb40881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 415 OBSOLETE_local: false peers { permanent_uuid: "84c5917a503d4a82bb2d9c0d5fdb04bd" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 32877 } } peers { permanent_uuid: "4e03e0d09b974c7d8cf7f49652d4fd23" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 43227 } } peers { permanent_uuid: "f74cb76209d04397a8a206f86e91ac5b" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 37091 } } peers { permanent_uuid: "2f6d2177839a452eba5e2fc39cb40881" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 35589 } attrs { promote: false } } peers { permanent_uuid: "c82fcb40e5534742981e8300df76a1f7" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 37067 } attrs { promote: false } }
I20240501 14:01:15.484704 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 26131
I20240501 14:01:15.531872 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 28530
I20240501 14:01:15.540608 28656 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:15.724059 29720 raft_consensus.cc:482] T 3dfc84e5be47406f95b6246252e8d0f5 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e03e0d09b974c7d8cf7f49652d4fd23)
I20240501 14:01:15.724336 29719 raft_consensus.cc:482] T f64ab2203c9042989dbcab79bab164a6 P c82fcb40e5534742981e8300df76a1f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 491e21cc623d4879b09975ede207ab2d)
I20240501 14:01:15.724493 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 28530
I20240501 14:01:15.752647 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 24424
I20240501 14:01:15.753729 24484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:15.939553 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 24424
2024-05-01T14:01:15Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (104301 ms)
[----------] 1 test from RebalanceParamTest (104301 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240501 14:01:16.097553 22551 test_util.cc:268] Using random seed: -343302158
I20240501 14:01:16.099134 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:01:16.099201 22551 ts_itest-base.cc:113] --------------
I20240501 14:01:16.099234 22551 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:01:16.099270 22551 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:01:16.099305 22551 ts_itest-base.cc:116] --------------
2024-05-01T14:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:01:16Z Disabled control of system clock
I20240501 14:01:16.160854 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:36941
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:36941
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:01:16.510138 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:16.510433 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:16.510603 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:16.518035 29729 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:16.518172 29729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:01:16.518249 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:16.518319 29729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:01:16.518381 29729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:01:16.527585 29729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:36941
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:16.528944 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:16.530742 29729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:16.541642 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:16.542210 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:16.542632 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:16.543292 29729 server_base.cc:1034] running on GCE node
I20240501 14:01:16.544099 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:16.545614 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:16.546851 29729 hybrid_clock.cc:648] HybridClock initialized: now 1714572076546799 us; error 47 us; skew 500 ppm
I20240501 14:01:16.547207 29729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:16.549499 29729 webserver.cc:460] Webserver started at http://127.22.5.254:41395/ using document root <none> and password file <none>
I20240501 14:01:16.550235 29729 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:16.550351 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:16.550681 29729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:16.553388 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "15344f00f71e4ebc8023c865b2536f99"
format_stamp: "Formatted at 2024-05-01 14:01:16 on dist-test-slave-5jzj"
I20240501 14:01:16.554499 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "15344f00f71e4ebc8023c865b2536f99"
format_stamp: "Formatted at 2024-05-01 14:01:16 on dist-test-slave-5jzj"
I20240501 14:01:16.561662 29729 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240501 14:01:16.565050 29743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:16.566437 29729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:16.566602 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "15344f00f71e4ebc8023c865b2536f99"
format_stamp: "Formatted at 2024-05-01 14:01:16 on dist-test-slave-5jzj"
I20240501 14:01:16.566802 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:16.593109 29729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:16.594049 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:16.594321 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:16.612344 29729 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:36941
I20240501 14:01:16.612362 29794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:36941 every 8 connection(s)
I20240501 14:01:16.613838 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:01:16.616971 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 29729
I20240501 14:01:16.617203 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:01:16.618218 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:16.630853 29795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99: Bootstrap starting.
I20240501 14:01:16.633883 29795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:16.634871 29795 log.cc:824] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:16.637578 29795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99: No bootstrap required, opened a new log
I20240501 14:01:16.643883 29795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } }
I20240501 14:01:16.644263 29795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:16.644366 29795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15344f00f71e4ebc8023c865b2536f99, State: Initialized, Role: FOLLOWER
I20240501 14:01:16.644989 29795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } }
I20240501 14:01:16.645255 29795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:01:16.645417 29795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:01:16.645601 29795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:16.647769 29795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } }
I20240501 14:01:16.648298 29795 leader_election.cc:302] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15344f00f71e4ebc8023c865b2536f99; no voters:
I20240501 14:01:16.648921 29795 leader_election.cc:288] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:01:16.649080 29800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:16.649755 29800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [term 1 LEADER]: Becoming Leader. State: Replica: 15344f00f71e4ebc8023c865b2536f99, State: Running, Role: LEADER
I20240501 14:01:16.650327 29800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } }
I20240501 14:01:16.651006 29795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:01:16.653394 29801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15344f00f71e4ebc8023c865b2536f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } } }
I20240501 14:01:16.653760 29801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:16.655015 29802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15344f00f71e4ebc8023c865b2536f99. Latest consensus state: current_term: 1 leader_uuid: "15344f00f71e4ebc8023c865b2536f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15344f00f71e4ebc8023c865b2536f99" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 36941 } } }
I20240501 14:01:16.655346 29802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:16.656739 29807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:01:16.662207 29807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:01:16.671541 29807 catalog_manager.cc:1347] Generated new cluster ID: 4e44780dceee42eb8e4a886f79c4e35c
I20240501 14:01:16.671676 29807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:01:16.690263 29807 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:01:16.691524 29807 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:01:16.711236 29807 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 15344f00f71e4ebc8023c865b2536f99: Generated new TSK 0
I20240501 14:01:16.712129 29807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:01:16.722385 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:01:17.078752 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:17.079051 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:17.079155 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:17.086514 29819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:17.086644 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:17.086788 29819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 14:01:17.095650 29819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:17.096946 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:17.098932 29819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:17.110922 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:17.111367 29827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:17.111855 29825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:17.112545 29819 server_base.cc:1034] running on GCE node
I20240501 14:01:17.113217 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:17.114178 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:17.115392 29819 hybrid_clock.cc:648] HybridClock initialized: now 1714572077115344 us; error 38 us; skew 500 ppm
I20240501 14:01:17.115753 29819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:17.118088 29819 webserver.cc:460] Webserver started at http://127.22.5.193:44179/ using document root <none> and password file <none>
I20240501 14:01:17.118881 29819 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:17.118992 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:17.119300 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:17.121984 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7869901e77234aeeb1462876b63eb28e"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.122732 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7869901e77234aeeb1462876b63eb28e"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.127761 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:01:17.131381 29833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.132861 29819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:17.133008 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7869901e77234aeeb1462876b63eb28e"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.133251 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:17.149868 29819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:17.150703 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:17.150956 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:17.151861 29819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:17.153375 29819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:17.153477 29819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.153580 29819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:17.153647 29819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.185151 29819 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:35665
I20240501 14:01:17.185200 29945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:35665 every 8 connection(s)
I20240501 14:01:17.186455 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:01:17.190198 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 29819
I20240501 14:01:17.190419 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:01:17.194120 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:17.200667 29946 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:17.201025 29946 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:17.202160 29946 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:17.204520 29760 ts_manager.cc:194] Registered new tserver with Master: 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:17.206439 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:44619
W20240501 14:01:17.545208 29950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:17.545511 29950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:17.545612 29950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:17.553200 29950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:17.553328 29950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:17.553458 29950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 14:01:17.562712 29950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:17.564029 29950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:17.565763 29950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:17.577927 29955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:17.578394 29956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:17.578853 29958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:17.579437 29950 server_base.cc:1034] running on GCE node
I20240501 14:01:17.580088 29950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:17.581053 29950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:17.582290 29950 hybrid_clock.cc:648] HybridClock initialized: now 1714572077582230 us; error 52 us; skew 500 ppm
I20240501 14:01:17.582620 29950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:17.584915 29950 webserver.cc:460] Webserver started at http://127.22.5.194:35773/ using document root <none> and password file <none>
I20240501 14:01:17.585741 29950 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:17.585858 29950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:17.586181 29950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:17.588862 29950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.589704 29950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.595085 29950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:01:17.598754 29964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.600129 29950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:17.600293 29950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
format_stamp: "Formatted at 2024-05-01 14:01:17 on dist-test-slave-5jzj"
I20240501 14:01:17.600504 29950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:17.634627 29950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:17.635514 29950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:17.635784 29950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:17.636759 29950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:17.638437 29950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:17.638554 29950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.638662 29950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:17.638731 29950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:17.671517 29950 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:38885
I20240501 14:01:17.671592 30076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:38885 every 8 connection(s)
I20240501 14:01:17.672904 29950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:01:17.676421 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 29950
I20240501 14:01:17.676622 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:01:17.680007 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:17.686340 30077 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:17.686672 30077 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:17.687525 30077 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:17.688969 29760 ts_manager.cc:194] Registered new tserver with Master: 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:17.689755 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:59949
W20240501 14:01:18.031652 30080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:18.031961 30080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:18.032065 30080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:18.039700 30080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:18.039829 30080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:18.039956 30080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:01:18.049093 30080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:18.050518 30080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:18.052313 30080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:18.064561 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:18.065061 30088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:18.065582 30086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:18.066195 30080 server_base.cc:1034] running on GCE node
I20240501 14:01:18.066812 30080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:18.067767 30080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:18.068964 30080 hybrid_clock.cc:648] HybridClock initialized: now 1714572078068914 us; error 48 us; skew 500 ppm
I20240501 14:01:18.069295 30080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:18.071557 30080 webserver.cc:460] Webserver started at http://127.22.5.195:33671/ using document root <none> and password file <none>
I20240501 14:01:18.072312 30080 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:18.072425 30080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:18.072759 30080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:18.075436 30080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "76c7a691de8a4e31b32799f43cb4b812"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.076283 30080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "76c7a691de8a4e31b32799f43cb4b812"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.081737 30080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:01:18.085273 30094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.086871 30080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:18.087054 30080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "76c7a691de8a4e31b32799f43cb4b812"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.087262 30080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:18.111049 30080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:18.111958 30080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:18.112228 30080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:18.113319 30080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:18.114890 30080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:18.114989 30080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.115097 30080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:18.115164 30080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.146992 30080 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:38823
I20240501 14:01:18.147037 30206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:38823 every 8 connection(s)
I20240501 14:01:18.148420 30080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:01:18.158149 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30080
I20240501 14:01:18.158362 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:01:18.160713 30207 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:18.161051 30207 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:18.161752 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:18.162199 30207 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:18.163918 29760 ts_manager.cc:194] Registered new tserver with Master: 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:18.164692 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:51657
I20240501 14:01:18.210057 29946 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
W20240501 14:01:18.512362 30210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:18.512681 30210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:18.512782 30210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:18.520239 30210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:18.520360 30210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:18.520491 30210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 14:01:18.529800 30210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:18.531203 30210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:18.532935 30210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:18.545521 30215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:18.546574 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:18.546654 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:18.547050 30210 server_base.cc:1034] running on GCE node
I20240501 14:01:18.547696 30210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:18.548638 30210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:18.549841 30210 hybrid_clock.cc:648] HybridClock initialized: now 1714572078549781 us; error 58 us; skew 500 ppm
I20240501 14:01:18.550149 30210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:18.552471 30210 webserver.cc:460] Webserver started at http://127.22.5.196:46877/ using document root <none> and password file <none>
I20240501 14:01:18.553262 30210 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:18.553385 30210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:18.553705 30210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:18.556358 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4a15e881772b41eab8a71fc027390db9"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.557207 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4a15e881772b41eab8a71fc027390db9"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.562669 30210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 14:01:18.566241 30224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.567657 30210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:18.567836 30210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4a15e881772b41eab8a71fc027390db9"
format_stamp: "Formatted at 2024-05-01 14:01:18 on dist-test-slave-5jzj"
I20240501 14:01:18.568039 30210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:18.583258 30210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:18.584112 30210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:18.584370 30210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:18.585314 30210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:18.586855 30210 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:18.586951 30210 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.587061 30210 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:18.587132 30210 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:18.619349 30210 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:39425
I20240501 14:01:18.619433 30336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:39425 every 8 connection(s)
I20240501 14:01:18.620832 30210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:01:18.629328 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30210
I20240501 14:01:18.629547 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:01:18.632927 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.197:0
--local_ip_for_outbound_sockets=127.22.5.197
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:18.633292 30337 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:18.633589 30337 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:18.634406 30337 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:18.635845 29760 ts_manager.cc:194] Registered new tserver with Master: 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:18.636636 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:57777
I20240501 14:01:18.692242 30077 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
W20240501 14:01:18.979614 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:18.979907 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:18.980053 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:18.987591 30340 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:18.987716 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:18.987893 30340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.197
I20240501 14:01:18.997023 30340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.5.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.197
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:18.998437 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:19.000113 30340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:19.012264 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:19.012841 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:19.013362 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:19.014025 30340 server_base.cc:1034] running on GCE node
I20240501 14:01:19.014622 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:19.015548 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:19.016745 30340 hybrid_clock.cc:648] HybridClock initialized: now 1714572079016703 us; error 41 us; skew 500 ppm
I20240501 14:01:19.017053 30340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:19.019330 30340 webserver.cc:460] Webserver started at http://127.22.5.197:35409/ using document root <none> and password file <none>
I20240501 14:01:19.020079 30340 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:19.020187 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:19.020514 30340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:19.023195 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.023967 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.029281 30340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 14:01:19.032691 30354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.034042 30340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:19.034190 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.034386 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:19.065490 30340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:19.066354 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:19.066630 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:19.067571 30340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:19.069072 30340 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:19.069190 30340 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.069304 30340 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:19.069375 30340 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.109323 30466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.197:34471 every 8 connection(s)
I20240501 14:01:19.109622 30340 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.197:34471
I20240501 14:01:19.111354 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:01:19.119755 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30340
I20240501 14:01:19.119983 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:01:19.124092 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.198:0
--local_ip_for_outbound_sockets=127.22.5.198
--webserver_interface=127.22.5.198
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:19.127066 30467 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:19.127532 30467 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:19.128760 30467 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:19.130856 29760 ts_manager.cc:194] Registered new tserver with Master: be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
I20240501 14:01:19.132126 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.197:33785
I20240501 14:01:19.167788 30207 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
W20240501 14:01:19.584964 30470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:19.585368 30470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:19.585650 30470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:19.593292 30470 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:19.593408 30470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:19.593591 30470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.198
I20240501 14:01:19.602715 30470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.5.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.198
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:19.604055 30470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:19.605813 30470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:19.618695 30476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:19.619370 30478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:19.619622 30475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:19.620954 30470 server_base.cc:1034] running on GCE node
I20240501 14:01:19.621646 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:19.622711 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:19.623930 30470 hybrid_clock.cc:648] HybridClock initialized: now 1714572079623883 us; error 34 us; skew 500 ppm
I20240501 14:01:19.624251 30470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:19.626816 30470 webserver.cc:460] Webserver started at http://127.22.5.198:46557/ using document root <none> and password file <none>
I20240501 14:01:19.627563 30470 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:19.627691 30470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:19.628031 30470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:19.630730 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3321c47e58ba49f09f4f036ac72fc133"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.631489 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3321c47e58ba49f09f4f036ac72fc133"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.636831 30470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:01:19.639266 30337 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:19.640671 30484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.642231 30470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:19.642416 30470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3321c47e58ba49f09f4f036ac72fc133"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:19.642614 30470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:19.660143 30470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:19.661015 30470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:19.661311 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:19.662319 30470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:19.663833 30470 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:19.663939 30470 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.664058 30470 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:19.664130 30470 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:19.696731 30470 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.198:36301
I20240501 14:01:19.696784 30596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.198:36301 every 8 connection(s)
I20240501 14:01:19.698146 30470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:01:19.703341 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30470
I20240501 14:01:19.703506 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:01:19.706919 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.199:0
--local_ip_for_outbound_sockets=127.22.5.199
--webserver_interface=127.22.5.199
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:19.711927 30597 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:19.712270 30597 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:19.713364 30597 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:19.714954 29760 ts_manager.cc:194] Registered new tserver with Master: 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
I20240501 14:01:19.715790 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.198:55999
W20240501 14:01:20.061388 30601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:20.061704 30601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:20.061877 30601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:20.069569 30601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:20.069725 30601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:20.069890 30601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.199
I20240501 14:01:20.079138 30601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.5.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.199
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:20.080569 30601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:20.082446 30601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:20.095263 30609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:20.095726 30607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:20.096203 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:20.096858 30601 server_base.cc:1034] running on GCE node
I20240501 14:01:20.097572 30601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:20.098582 30601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:20.099820 30601 hybrid_clock.cc:648] HybridClock initialized: now 1714572080099766 us; error 41 us; skew 500 ppm
I20240501 14:01:20.100175 30601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:20.103088 30601 webserver.cc:460] Webserver started at http://127.22.5.199:39397/ using document root <none> and password file <none>
I20240501 14:01:20.103909 30601 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:20.104022 30601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:20.104426 30601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:20.107034 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.107774 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.112978 30601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 14:01:20.116423 30615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.117797 30601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:20.117949 30601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.118144 30601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:20.133215 30601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:20.134110 30601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:20.134351 30601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:20.135298 30601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:20.135538 30467 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:20.136909 30601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:20.137009 30601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.137159 30601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:20.137243 30601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.169008 30601 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.199:40447
I20240501 14:01:20.169056 30727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.199:40447 every 8 connection(s)
I20240501 14:01:20.170370 30601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:01:20.175647 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30601
I20240501 14:01:20.175817 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:01:20.178798 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.200:0
--local_ip_for_outbound_sockets=127.22.5.200
--webserver_interface=127.22.5.200
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:20.187963 30728 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:20.188297 30728 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:20.189191 30728 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:20.190753 29760 ts_manager.cc:194] Registered new tserver with Master: 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447)
I20240501 14:01:20.191728 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.199:40075
W20240501 14:01:20.541823 30731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:20.542143 30731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:20.542290 30731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:20.550125 30731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:20.550244 30731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:20.550448 30731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.200
I20240501 14:01:20.559746 30731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.5.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.200
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:20.561182 30731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:20.562950 30731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:20.575513 30739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:20.576018 30736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:20.576478 30737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:20.577149 30731 server_base.cc:1034] running on GCE node
I20240501 14:01:20.577790 30731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:20.578856 30731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:20.580075 30731 hybrid_clock.cc:648] HybridClock initialized: now 1714572080580025 us; error 42 us; skew 500 ppm
I20240501 14:01:20.580371 30731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:20.585656 30731 webserver.cc:460] Webserver started at http://127.22.5.200:42041/ using document root <none> and password file <none>
I20240501 14:01:20.586409 30731 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:20.586506 30731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:20.586782 30731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:20.589498 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "744cef852ee9491db3928fa0092ef189"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.590255 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "744cef852ee9491db3928fa0092ef189"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.595592 30731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:01:20.598970 30745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.600214 30731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:20.600363 30731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "744cef852ee9491db3928fa0092ef189"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:20.600555 30731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:20.641958 30731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:20.642877 30731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:20.643139 30731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:20.644074 30731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:20.645637 30731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:20.645735 30731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.645843 30731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:20.645915 30731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:20.677320 30731 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.200:43981
I20240501 14:01:20.677374 30857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.200:43981 every 8 connection(s)
I20240501 14:01:20.678601 30731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:01:20.687175 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 30731
I20240501 14:01:20.687357 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:01:20.692533 30858 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:20.692843 30858 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:20.693771 30858 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:20.695071 29760 ts_manager.cc:194] Registered new tserver with Master: 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:01:20.695838 29760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.200:34285
I20240501 14:01:20.704507 22551 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:01:20.718835 30597 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:20.720670 22551 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:01:20.720791 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30340
I20240501 14:01:20.728628 30462 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:20.809450 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30340
I20240501 14:01:20.828138 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30470
I20240501 14:01:20.835906 30592 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:20.917371 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30470
I20240501 14:01:20.935146 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30601
I20240501 14:01:20.943331 30723 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:21.022490 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30601
I20240501 14:01:21.039325 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30731
I20240501 14:01:21.047287 30853 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:21.127871 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30731
I20240501 14:01:24.896066 29759 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:24.940276 29881 tablet_service.cc:1467] Processing CreateTablet for tablet a733857ad51d452a9054cc5f533edc8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.940276 30142 tablet_service.cc:1467] Processing CreateTablet for tablet a733857ad51d452a9054cc5f533edc8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.940263 30272 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6d21ea3184a6eb9db2b87589781b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.941159 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 5c35ccca22fb45b78ee66991027c0261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.941646 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 5c35ccca22fb45b78ee66991027c0261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.941867 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a733857ad51d452a9054cc5f533edc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.941890 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a733857ad51d452a9054cc5f533edc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.942095 29880 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6d21ea3184a6eb9db2b87589781b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.942476 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c35ccca22fb45b78ee66991027c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.942531 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d21ea3184a6eb9db2b87589781b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.943945 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 5c35ccca22fb45b78ee66991027c0261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.944442 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c35ccca22fb45b78ee66991027c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.945144 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c35ccca22fb45b78ee66991027c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.947432 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d21ea3184a6eb9db2b87589781b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.961270 30887 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:24.962076 30888 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:24.962980 30889 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:24.966114 30011 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6d21ea3184a6eb9db2b87589781b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.967775 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d21ea3184a6eb9db2b87589781b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.968164 30887 tablet_bootstrap.cc:654] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.969080 30889 tablet_bootstrap.cc:654] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.969195 30887 log.cc:824] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:24.970254 30889 log.cc:824] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:24.970618 30888 tablet_bootstrap.cc:654] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.972241 30888 log.cc:824] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:24.966085 30012 tablet_service.cc:1467] Processing CreateTablet for tablet a733857ad51d452a9054cc5f533edc8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:24.973013 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a733857ad51d452a9054cc5f533edc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.979099 30887 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:24.979482 30887 ts_tablet_manager.cc:1367] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20240501 14:01:24.988176 30889 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:24.988575 30889 ts_tablet_manager.cc:1367] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.012s
I20240501 14:01:24.991370 30888 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:24.991722 30888 ts_tablet_manager.cc:1367] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.008s
I20240501 14:01:24.993587 30893 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:24.994060 30887 raft_consensus.cc:348] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:24.994614 30887 raft_consensus.cc:374] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.994752 30887 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.995615 30887 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:24.997526 30889 raft_consensus.cc:348] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:24.997828 30893 tablet_bootstrap.cc:654] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.998100 30889 raft_consensus.cc:374] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.998242 30889 raft_consensus.cc:729] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.999076 30889 consensus_queue.cc:260] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:24.999429 30893 log.cc:824] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:25.000689 30888 raft_consensus.cc:348] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.001267 30888 raft_consensus.cc:374] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.001430 30888 raft_consensus.cc:729] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.002279 30888 consensus_queue.cc:260] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.005075 30887 ts_tablet_manager.cc:1398] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.025s user 0.014s sys 0.007s
I20240501 14:01:25.005555 30887 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:25.007248 30887 tablet_bootstrap.cc:654] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:25.008034 30888 ts_tablet_manager.cc:1398] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20240501 14:01:25.010782 30888 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:25.013016 30888 tablet_bootstrap.cc:654] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:25.016708 30888 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:25.016911 30888 ts_tablet_manager.cc:1367] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 14:01:25.017845 30889 ts_tablet_manager.cc:1398] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s
I20240501 14:01:25.018016 30888 raft_consensus.cc:348] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.018229 30888 raft_consensus.cc:374] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.018328 30888 raft_consensus.cc:729] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.018545 30889 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:25.018563 30888 consensus_queue.cc:260] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.019150 30888 ts_tablet_manager.cc:1398] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:25.020391 30889 tablet_bootstrap.cc:654] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:25.021881 30887 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:25.022046 30887 ts_tablet_manager.cc:1367] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240501 14:01:25.022368 30893 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:25.022663 30893 ts_tablet_manager.cc:1367] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.005s
I20240501 14:01:25.023069 30887 raft_consensus.cc:348] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.023320 30887 raft_consensus.cc:374] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.023438 30887 raft_consensus.cc:729] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.023713 30887 consensus_queue.cc:260] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.024886 30887 ts_tablet_manager.cc:1398] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:01:25.028756 30889 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:25.028887 30889 ts_tablet_manager.cc:1367] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240501 14:01:25.029722 30889 raft_consensus.cc:348] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.029924 30889 raft_consensus.cc:374] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.029990 30889 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.030186 30889 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.030016 30893 raft_consensus.cc:348] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.030428 30893 raft_consensus.cc:374] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.030541 30893 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.030624 30889 ts_tablet_manager.cc:1398] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:25.030882 30889 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:25.031239 30893 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.032474 30889 tablet_bootstrap.cc:654] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:25.034037 30893 ts_tablet_manager.cc:1398] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20240501 14:01:25.034523 30893 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:25.036098 30893 tablet_bootstrap.cc:654] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:25.038240 30893 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:25.038272 30889 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:25.038383 30893 ts_tablet_manager.cc:1367] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:01:25.038462 30889 ts_tablet_manager.cc:1367] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 14:01:25.039009 30889 raft_consensus.cc:348] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.039193 30889 raft_consensus.cc:374] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.039278 30889 raft_consensus.cc:729] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.039211 30893 raft_consensus.cc:348] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.039399 30893 raft_consensus.cc:374] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:25.039499 30893 raft_consensus.cc:729] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:25.039526 30889 consensus_queue.cc:260] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.039700 30893 consensus_queue.cc:260] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.040059 30889 ts_tablet_manager.cc:1398] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:25.040249 30893 ts_tablet_manager.cc:1398] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240501 14:01:25.126780 30338 tablet.cc:2369] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:01:25.155108 30208 tablet.cc:2369] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:25.167192 30895 raft_consensus.cc:482] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:25.167448 30895 raft_consensus.cc:504] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.169049 30895 leader_election.cc:288] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:25.178277 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:25.178686 30032 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:25.179229 29837 leader_election.cc:302] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:25.179327 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:25.179651 30895 raft_consensus.cc:2781] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:25.179785 30895 raft_consensus.cc:482] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:25.179769 30292 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:25.179874 30895 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:25.180063 30078 tablet.cc:2369] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:25.182256 30906 raft_consensus.cc:482] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:25.182346 30895 raft_consensus.cc:504] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.182430 30906 raft_consensus.cc:504] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.183020 30895 leader_election.cc:288] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:25.183353 30906 leader_election.cc:288] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:25.183471 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:25.183640 30292 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.183696 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:25.183861 30032 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.186722 30292 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:25.186722 30032 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:25.187424 30291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c35ccca22fb45b78ee66991027c0261" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:25.187687 30291 raft_consensus.cc:2454] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:25.187964 29836 leader_election.cc:302] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:25.188247 29836 leader_election.cc:302] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:25.188333 30906 raft_consensus.cc:2781] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:25.189179 30906 raft_consensus.cc:686] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:25.189347 30907 raft_consensus.cc:2781] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:25.189476 30907 raft_consensus.cc:482] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:25.189620 30907 raft_consensus.cc:3037] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.189939 30906 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:25.190887 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c35ccca22fb45b78ee66991027c0261" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:25.191366 30162 raft_consensus.cc:2454] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:25.192524 30907 raft_consensus.cc:504] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.193145 30907 leader_election.cc:288] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:25.193493 30291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c35ccca22fb45b78ee66991027c0261" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:25.193670 30291 raft_consensus.cc:3037] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.194008 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c35ccca22fb45b78ee66991027c0261" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:25.194166 30162 raft_consensus.cc:3037] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.196506 30291 raft_consensus.cc:2454] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:25.197078 30162 raft_consensus.cc:2454] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
W20240501 14:01:25.193548 29947 tablet.cc:2369] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:25.196888 29836 leader_election.cc:302] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:25.197450 30895 raft_consensus.cc:2781] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:25.197593 30895 raft_consensus.cc:686] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:25.197844 30895 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.201072 29758 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:25.206156 29758 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:25.272617 30901 raft_consensus.cc:482] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:25.272850 30901 raft_consensus.cc:504] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.274590 30901 leader_election.cc:288] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:25.281257 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a733857ad51d452a9054cc5f533edc8e" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:25.281558 30162 raft_consensus.cc:2454] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:25.281592 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a733857ad51d452a9054cc5f533edc8e" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:25.282028 29901 raft_consensus.cc:2454] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:25.282043 29968 leader_election.cc:302] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:25.282493 30901 raft_consensus.cc:2781] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:25.282639 30901 raft_consensus.cc:482] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:25.282747 30901 raft_consensus.cc:3037] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.284732 30901 raft_consensus.cc:504] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.285358 30901 leader_election.cc:288] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:25.285734 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a733857ad51d452a9054cc5f533edc8e" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:25.285918 30162 raft_consensus.cc:3037] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.285874 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a733857ad51d452a9054cc5f533edc8e" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:25.286052 29901 raft_consensus.cc:3037] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:25.287746 30162 raft_consensus.cc:2454] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:25.288065 29901 raft_consensus.cc:2454] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:25.288244 29968 leader_election.cc:302] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:25.288748 30901 raft_consensus.cc:2781] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:25.289398 30901 raft_consensus.cc:686] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:25.289983 30901 consensus_queue.cc:237] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:25.294616 29758 catalog_manager.cc:5484] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:25.585912 30895 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:25.596797 30907 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:25.660538 30907 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:25.666843 30907 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:25.704421 30901 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:25.710299 30912 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:25.734299 30914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:25.734674 30914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:25.742532 30914 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 222.056 ms
time per row: 0.111028 ms
I20240501 14:01:26.112368 29759 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:26.121598 30141 tablet_service.cc:1467] Processing CreateTablet for tablet c40a79b118424a5699bddd1c4ac4bb0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.121812 30272 tablet_service.cc:1467] Processing CreateTablet for tablet c40a79b118424a5699bddd1c4ac4bb0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.122118 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a79b118424a5699bddd1c4ac4bb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.122174 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a79b118424a5699bddd1c4ac4bb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.122129 30011 tablet_service.cc:1467] Processing CreateTablet for tablet c40a79b118424a5699bddd1c4ac4bb0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.122582 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a79b118424a5699bddd1c4ac4bb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.123749 29881 tablet_service.cc:1467] Processing CreateTablet for tablet d1bde99cadd94efc826b4074b2674104 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.124163 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1bde99cadd94efc826b4074b2674104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.128901 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 642312cdad114fce839d2b60717bcc72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.129456 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642312cdad114fce839d2b60717bcc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.133214 30982 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:26.134091 30980 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:26.134708 30142 tablet_service.cc:1467] Processing CreateTablet for tablet 642312cdad114fce839d2b60717bcc72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.134742 30012 tablet_service.cc:1467] Processing CreateTablet for tablet 642312cdad114fce839d2b60717bcc72 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.134887 30010 tablet_service.cc:1467] Processing CreateTablet for tablet d1bde99cadd94efc826b4074b2674104 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.135123 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642312cdad114fce839d2b60717bcc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.135288 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642312cdad114fce839d2b60717bcc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.136139 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1bde99cadd94efc826b4074b2674104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.136876 30982 tablet_bootstrap.cc:654] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.138134 30140 tablet_service.cc:1467] Processing CreateTablet for tablet d1bde99cadd94efc826b4074b2674104 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:26.138576 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1bde99cadd94efc826b4074b2674104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.145841 30980 tablet_bootstrap.cc:654] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.148007 30979 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:26.150583 30979 tablet_bootstrap.cc:654] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.152081 30981 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:26.152434 30982 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:26.152853 30982 ts_tablet_manager.cc:1367] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240501 14:01:26.153824 30982 raft_consensus.cc:348] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.154235 30982 raft_consensus.cc:374] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.154408 30982 raft_consensus.cc:729] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.154892 30981 tablet_bootstrap.cc:654] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.154875 30982 consensus_queue.cc:260] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.155661 30982 ts_tablet_manager.cc:1398] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:26.159592 30979 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:26.159814 30979 ts_tablet_manager.cc:1367] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:01:26.160753 30980 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:26.160734 30979 raft_consensus.cc:348] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.160995 30979 raft_consensus.cc:374] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.161017 30980 ts_tablet_manager.cc:1367] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20240501 14:01:26.161157 30979 raft_consensus.cc:729] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.161507 30979 consensus_queue.cc:260] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.161972 30980 raft_consensus.cc:348] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.162232 30980 raft_consensus.cc:374] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.162376 30980 raft_consensus.cc:729] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.162714 30980 consensus_queue.cc:260] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.162945 30979 ts_tablet_manager.cc:1398] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:26.163444 30980 ts_tablet_manager.cc:1398] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:26.163764 30979 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:26.164997 30915 raft_consensus.cc:482] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:26.165242 30915 raft_consensus.cc:504] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.165501 30979 tablet_bootstrap.cc:654] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.167393 30915 leader_election.cc:288] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:26.171222 30981 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:26.172228 30981 ts_tablet_manager.cc:1367] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20240501 14:01:26.173172 30981 raft_consensus.cc:348] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.173590 30981 raft_consensus.cc:374] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.173713 30981 raft_consensus.cc:729] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.174000 30981 consensus_queue.cc:260] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.174263 30980 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:26.175089 30981 ts_tablet_manager.cc:1398] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:26.175402 30981 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:26.177650 30981 tablet_bootstrap.cc:654] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.182163 30980 tablet_bootstrap.cc:654] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.184330 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a79b118424a5699bddd1c4ac4bb0e" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:26.184557 30032 raft_consensus.cc:2454] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:26.185164 30228 leader_election.cc:302] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:26.185581 30915 raft_consensus.cc:2781] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:26.185727 30915 raft_consensus.cc:482] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:26.185840 30915 raft_consensus.cc:3037] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.188813 30161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a79b118424a5699bddd1c4ac4bb0e" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:26.189067 30161 raft_consensus.cc:2454] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:26.189471 30979 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:26.189596 30981 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:26.189633 30979 ts_tablet_manager.cc:1367] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.016s
I20240501 14:01:26.189769 30981 ts_tablet_manager.cc:1367] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240501 14:01:26.190102 30915 raft_consensus.cc:504] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.190416 30979 raft_consensus.cc:348] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.190469 30981 raft_consensus.cc:348] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.190658 30979 raft_consensus.cc:374] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.190701 30981 raft_consensus.cc:374] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.190742 30979 raft_consensus.cc:729] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.190768 30981 raft_consensus.cc:729] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.191048 30981 consensus_queue.cc:260] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.191066 30979 consensus_queue.cc:260] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.191533 30981 ts_tablet_manager.cc:1398] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:26.191737 30979 ts_tablet_manager.cc:1398] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:26.191854 30981 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:26.192039 30979 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:26.192657 30915 leader_election.cc:288] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:26.193403 30981 tablet_bootstrap.cc:654] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.193488 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a79b118424a5699bddd1c4ac4bb0e" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:26.193714 30032 raft_consensus.cc:3037] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.195456 30979 tablet_bootstrap.cc:654] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:26.195817 30980 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:26.196028 30980 ts_tablet_manager.cc:1367] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20240501 14:01:26.196995 30980 raft_consensus.cc:348] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.197413 30980 raft_consensus.cc:374] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.197564 30980 raft_consensus.cc:729] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.197849 30161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a79b118424a5699bddd1c4ac4bb0e" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:26.198038 30161 raft_consensus.cc:3037] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.197889 30980 consensus_queue.cc:260] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.198658 30980 ts_tablet_manager.cc:1398] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:26.199779 30032 raft_consensus.cc:2454] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:26.200788 30161 raft_consensus.cc:2454] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:26.201826 30979 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:26.202006 30979 ts_tablet_manager.cc:1367] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240501 14:01:26.202524 30228 leader_election.cc:302] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:26.202804 30979 raft_consensus.cc:348] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.202929 30915 raft_consensus.cc:2781] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:26.203018 30979 raft_consensus.cc:374] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.203107 30979 raft_consensus.cc:729] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.203332 30979 consensus_queue.cc:260] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.203897 30979 ts_tablet_manager.cc:1398] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:26.205881 30981 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:26.206058 30981 ts_tablet_manager.cc:1367] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240501 14:01:26.206825 30981 raft_consensus.cc:348] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.207054 30981 raft_consensus.cc:374] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:26.207163 30981 raft_consensus.cc:729] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:26.207738 30915 raft_consensus.cc:686] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:26.208330 30981 consensus_queue.cc:260] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.208313 30915 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:26.209403 30981 ts_tablet_manager.cc:1398] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:26.212081 29759 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:26.447221 30901 raft_consensus.cc:482] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:26.447427 30901 raft_consensus.cc:504] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.448035 30901 leader_election.cc:288] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:26.448606 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1bde99cadd94efc826b4074b2674104" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:26.448604 30161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1bde99cadd94efc826b4074b2674104" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:26.448839 29901 raft_consensus.cc:2454] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:26.448832 30161 raft_consensus.cc:2454] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:26.449286 29965 leader_election.cc:302] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:26.449569 30901 raft_consensus.cc:2781] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:26.449695 30901 raft_consensus.cc:482] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:26.449765 30901 raft_consensus.cc:3037] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.451656 30901 raft_consensus.cc:504] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.452180 30901 leader_election.cc:288] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:26.452582 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1bde99cadd94efc826b4074b2674104" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:26.452770 29901 raft_consensus.cc:3037] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.452718 30161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1bde99cadd94efc826b4074b2674104" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:26.452929 30161 raft_consensus.cc:3037] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.454633 29901 raft_consensus.cc:2454] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:26.454826 30161 raft_consensus.cc:2454] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:26.455062 29965 leader_election.cc:302] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:26.455368 30901 raft_consensus.cc:2781] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:26.455485 30901 raft_consensus.cc:686] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:26.455747 30901 consensus_queue.cc:237] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:26.458351 29759 catalog_manager.cc:5484] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:26.472458 30919 raft_consensus.cc:482] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:26.472663 30919 raft_consensus.cc:504] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.474316 30919 leader_election.cc:288] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:26.481496 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:26.481580 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:26.481734 30290 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:26.481808 30032 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:26.482228 30097 leader_election.cc:302] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:26.482589 30919 raft_consensus.cc:2781] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:26.482719 30919 raft_consensus.cc:482] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:26.482817 30919 raft_consensus.cc:3037] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.484786 30919 raft_consensus.cc:504] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.485368 30919 leader_election.cc:288] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:26.485781 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:26.485843 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:26.485987 30032 raft_consensus.cc:3037] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.486014 30290 raft_consensus.cc:3037] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:26.488817 30032 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:26.488832 30290 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:26.489243 30097 leader_election.cc:302] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:26.489534 30919 raft_consensus.cc:2781] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:26.489892 30919 raft_consensus.cc:686] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:26.490374 30919 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:26.493672 29759 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:26.597709 30915 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:26.604424 30915 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:26.892611 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:26.892993 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:26.900588 31003 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:01:26.989558 30901 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.006565 30901 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.049141 30919 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.051257 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:27.052645 30919 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.102131 31007 mvcc.cc:205] Tried to move back new op lower bound from 7022887268547641344 to 7022887266268225536. Current Snapshot: MvccSnapshot[applied={T|T < 7022887268547641344}]
INSERT report
rows total: 2000
time total: 242 ms
time per row: 0.121 ms
I20240501 14:01:27.290740 29759 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:27.299360 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 582bddb40ece4d5b8dcd3e1c00a53536 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.299711 29881 tablet_service.cc:1467] Processing CreateTablet for tablet 582bddb40ece4d5b8dcd3e1c00a53536 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.299861 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bddb40ece4d5b8dcd3e1c00a53536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.300112 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bddb40ece4d5b8dcd3e1c00a53536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.300839 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.301190 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d94ed11f0f4a9d9f519a24c3320ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.304951 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.305016 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.304993 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 582bddb40ece4d5b8dcd3e1c00a53536 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.305537 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.305636 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.305752 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bddb40ece4d5b8dcd3e1c00a53536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.306033 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d94ed11f0f4a9d9f519a24c3320ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.307571 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.307998 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d94ed11f0f4a9d9f519a24c3320ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.308964 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:27.309439 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.314143 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:27.326361 31072 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:27.329511 31072 tablet_bootstrap.cc:654] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.330157 31071 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:27.331763 31071 tablet_bootstrap.cc:654] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.334594 31071 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:27.334801 31071 ts_tablet_manager.cc:1367] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:27.335515 31071 raft_consensus.cc:348] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.335677 31071 raft_consensus.cc:374] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.335857 31071 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.336601 31072 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:27.336735 31072 ts_tablet_manager.cc:1367] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:01:27.337518 31072 raft_consensus.cc:348] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.337419 31071 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.337744 31073 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:27.337980 31072 raft_consensus.cc:374] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.338140 31072 raft_consensus.cc:729] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.338788 31071 ts_tablet_manager.cc:1398] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:27.339103 31071 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:27.340272 31072 consensus_queue.cc:260] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.340740 31073 tablet_bootstrap.cc:654] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.341740 31071 tablet_bootstrap.cc:654] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.343858 31070 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:27.345660 31073 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:27.345805 31070 tablet_bootstrap.cc:654] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.345881 31073 ts_tablet_manager.cc:1367] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240501 14:01:27.346251 31072 ts_tablet_manager.cc:1398] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:01:27.346793 31073 raft_consensus.cc:348] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.348624 31073 raft_consensus.cc:374] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.348804 31073 raft_consensus.cc:729] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.349167 31073 consensus_queue.cc:260] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.349468 31072 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:27.350035 31073 ts_tablet_manager.cc:1398] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:27.351563 31070 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:27.351588 31071 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:27.351702 31070 ts_tablet_manager.cc:1367] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240501 14:01:27.351799 31071 ts_tablet_manager.cc:1367] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240501 14:01:27.352463 31070 raft_consensus.cc:348] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.352687 31070 raft_consensus.cc:374] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.352623 31071 raft_consensus.cc:348] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.352787 31070 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.352856 31071 raft_consensus.cc:374] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.353029 31071 raft_consensus.cc:729] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.353040 31070 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.353319 31071 consensus_queue.cc:260] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.353868 31071 ts_tablet_manager.cc:1398] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:27.354570 31070 ts_tablet_manager.cc:1398] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:27.354848 31070 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:27.355540 31072 tablet_bootstrap.cc:654] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.356429 31070 tablet_bootstrap.cc:654] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.358723 31070 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:27.358948 31070 ts_tablet_manager.cc:1367] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:27.359627 31070 raft_consensus.cc:348] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.359786 31070 raft_consensus.cc:374] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.359853 31070 raft_consensus.cc:729] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.360072 31070 consensus_queue.cc:260] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.360489 31070 ts_tablet_manager.cc:1398] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:27.360771 31070 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:27.362265 31070 tablet_bootstrap.cc:654] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.365425 31070 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:27.365571 31070 ts_tablet_manager.cc:1367] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:01:27.366109 31070 raft_consensus.cc:348] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.366256 31070 raft_consensus.cc:374] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.366276 31072 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:27.366328 31070 raft_consensus.cc:729] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.366463 31072 ts_tablet_manager.cc:1367] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20240501 14:01:27.366523 31070 consensus_queue.cc:260] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.366976 31070 ts_tablet_manager.cc:1398] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:27.367695 31072 raft_consensus.cc:348] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.367930 31072 raft_consensus.cc:374] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.368013 31072 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.368249 31072 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.368729 31072 ts_tablet_manager.cc:1398] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:27.369038 31072 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:27.370396 31072 tablet_bootstrap.cc:654] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:27.372313 31072 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:27.372450 31072 ts_tablet_manager.cc:1367] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:27.373189 31072 raft_consensus.cc:348] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.373354 31072 raft_consensus.cc:374] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:27.373454 31072 raft_consensus.cc:729] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:27.373625 31072 consensus_queue.cc:260] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.374047 31072 ts_tablet_manager.cc:1398] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:27.420682 30895 raft_consensus.cc:482] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:27.420867 30895 raft_consensus.cc:504] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.421578 30895 leader_election.cc:288] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:27.422108 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:27.422135 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:27.422364 30032 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:27.422348 30290 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:27.422783 29836 leader_election.cc:302] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:27.423039 30895 raft_consensus.cc:2781] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:27.423157 30895 raft_consensus.cc:482] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:27.423269 30895 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.425191 30895 raft_consensus.cc:504] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.425781 30895 leader_election.cc:288] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:27.426256 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:27.426262 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:27.426443 30290 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.426446 30032 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.428334 30290 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:27.428335 30032 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:27.428763 29837 leader_election.cc:302] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:27.429050 30895 raft_consensus.cc:2781] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:27.429468 30895 raft_consensus.cc:686] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:27.429773 30895 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.432510 29760 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:27.487197 30895 raft_consensus.cc:482] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:27.487429 30895 raft_consensus.cc:504] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.488139 30895 leader_election.cc:288] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:27.488716 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccf3fc5fc7f44a3bc53dad7902c2d3a" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:27.488773 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccf3fc5fc7f44a3bc53dad7902c2d3a" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:27.488962 30290 raft_consensus.cc:2454] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:27.488984 30032 raft_consensus.cc:2454] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:27.489454 29837 leader_election.cc:302] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:27.489921 30895 raft_consensus.cc:2781] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:27.490039 30895 raft_consensus.cc:482] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:27.490175 30895 raft_consensus.cc:3037] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.492352 30895 raft_consensus.cc:504] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.493103 30895 leader_election.cc:288] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:27.493598 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccf3fc5fc7f44a3bc53dad7902c2d3a" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:27.493822 30290 raft_consensus.cc:3037] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.493747 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccf3fc5fc7f44a3bc53dad7902c2d3a" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:27.493975 30032 raft_consensus.cc:3037] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.495837 30290 raft_consensus.cc:2454] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:27.496044 30032 raft_consensus.cc:2454] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:27.496344 29836 leader_election.cc:302] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:27.496660 30895 raft_consensus.cc:2781] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:27.496834 30895 raft_consensus.cc:686] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:27.497107 30895 consensus_queue.cc:237] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:27.499905 29760 catalog_manager.cc:5484] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:27.574944 30919 raft_consensus.cc:482] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:27.575189 30919 raft_consensus.cc:504] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.575986 30919 leader_election.cc:288] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:27.576649 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d94ed11f0f4a9d9f519a24c3320ce4" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:27.576928 30290 raft_consensus.cc:2454] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:27.577432 30097 leader_election.cc:302] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:27.577797 30919 raft_consensus.cc:2781] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:27.577912 30919 raft_consensus.cc:482] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:27.578028 30919 raft_consensus.cc:3037] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.579958 30919 raft_consensus.cc:504] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.580632 30919 leader_election.cc:288] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:27.581183 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d94ed11f0f4a9d9f519a24c3320ce4" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:27.581444 30290 raft_consensus.cc:3037] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.582523 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d94ed11f0f4a9d9f519a24c3320ce4" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:27.582609 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d94ed11f0f4a9d9f519a24c3320ce4" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:27.582798 29900 raft_consensus.cc:2454] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:27.583928 30290 raft_consensus.cc:2454] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:27.584324 30097 leader_election.cc:302] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters: 7869901e77234aeeb1462876b63eb28e
I20240501 14:01:27.584601 30919 raft_consensus.cc:2781] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:27.584743 30919 raft_consensus.cc:686] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:27.584962 30919 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:27.587584 29760 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:27.883194 30895 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.889819 30895 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.929996 30895 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:27.935770 30895 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:27.970590 31089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:27.970901 31089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:27.979621 30919 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:27.979766 31089 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:27.984643 29901 raft_consensus.cc:3037] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:27.987715 30919 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 207.291 ms
time per row: 0.103645 ms
I20240501 14:01:28.333339 29760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:28.342311 29881 tablet_service.cc:1467] Processing CreateTablet for tablet ef202920b8d64287a8a2cdb40c73a819 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.342504 30141 tablet_service.cc:1467] Processing CreateTablet for tablet ef202920b8d64287a8a2cdb40c73a819 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.342847 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef202920b8d64287a8a2cdb40c73a819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.342952 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef202920b8d64287a8a2cdb40c73a819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.343125 29880 tablet_service.cc:1467] Processing CreateTablet for tablet c953ab40905a4d23b9b9c970d25168b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.343492 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c953ab40905a4d23b9b9c970d25168b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.344250 30271 tablet_service.cc:1467] Processing CreateTablet for tablet c953ab40905a4d23b9b9c970d25168b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.344682 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c953ab40905a4d23b9b9c970d25168b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.346200 30140 tablet_service.cc:1467] Processing CreateTablet for tablet c953ab40905a4d23b9b9c970d25168b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.346557 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c953ab40905a4d23b9b9c970d25168b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.348183 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.348527 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56df9dcda34e6a8d27a8a6b24742b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.349494 30142 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.349843 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56df9dcda34e6a8d27a8a6b24742b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.350531 30011 tablet_service.cc:1467] Processing CreateTablet for tablet ef202920b8d64287a8a2cdb40c73a819 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.350927 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef202920b8d64287a8a2cdb40c73a819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.356758 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:28.357144 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56df9dcda34e6a8d27a8a6b24742b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:28.359386 31152 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:28.363943 31151 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:28.364499 31150 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:28.364548 31152 tablet_bootstrap.cc:654] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.366406 31151 tablet_bootstrap.cc:654] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.366537 31150 tablet_bootstrap.cc:654] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.368155 31154 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:28.370782 31154 tablet_bootstrap.cc:654] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.371187 31151 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:28.372555 31152 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:28.371364 31151 ts_tablet_manager.cc:1367] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 14:01:28.372704 31152 ts_tablet_manager.cc:1367] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20240501 14:01:28.373581 31150 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:28.373625 31152 raft_consensus.cc:348] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.373625 31151 raft_consensus.cc:348] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.373818 31150 ts_tablet_manager.cc:1367] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 14:01:28.373884 31152 raft_consensus.cc:374] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.373937 31151 raft_consensus.cc:374] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.374019 31152 raft_consensus.cc:729] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.374104 31151 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.374341 31152 consensus_queue.cc:260] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.374403 31151 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.374882 31150 raft_consensus.cc:348] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.375216 31150 raft_consensus.cc:374] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.375347 31150 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.375633 31150 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.376993 31150 ts_tablet_manager.cc:1398] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:28.377422 31150 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:28.377785 31152 ts_tablet_manager.cc:1398] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:28.378715 31152 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:28.379328 31150 tablet_bootstrap.cc:654] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.380741 31152 tablet_bootstrap.cc:654] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.381563 31151 ts_tablet_manager.cc:1398] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:01:28.382707 31154 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:28.382974 31154 ts_tablet_manager.cc:1367] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240501 14:01:28.384907 31154 raft_consensus.cc:348] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.385318 31154 raft_consensus.cc:374] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.385373 31151 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:28.385504 31154 raft_consensus.cc:729] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.385862 31154 consensus_queue.cc:260] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.387437 31151 tablet_bootstrap.cc:654] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.387539 31154 ts_tablet_manager.cc:1398] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:28.387993 31154 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:28.392776 31154 tablet_bootstrap.cc:654] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.393309 31152 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:28.393604 31152 ts_tablet_manager.cc:1367] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.002s
I20240501 14:01:28.394695 31152 raft_consensus.cc:348] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.395068 31152 raft_consensus.cc:374] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.395202 31152 raft_consensus.cc:729] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.395579 31152 consensus_queue.cc:260] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.396546 31151 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:28.396773 31151 ts_tablet_manager.cc:1367] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240501 14:01:28.397212 31150 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:28.397451 31150 ts_tablet_manager.cc:1367] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240501 14:01:28.397691 31151 raft_consensus.cc:348] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.397987 31151 raft_consensus.cc:374] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.398115 31151 raft_consensus.cc:729] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.398320 30915 raft_consensus.cc:482] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:28.398475 30915 raft_consensus.cc:504] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.398439 31150 raft_consensus.cc:348] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.398550 31151 consensus_queue.cc:260] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.398869 31150 raft_consensus.cc:374] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.398993 31150 raft_consensus.cc:729] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.399129 30915 leader_election.cc:288] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:28.399335 31150 consensus_queue.cc:260] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.399848 31154 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:28.400000 31150 ts_tablet_manager.cc:1398] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:28.400056 31152 ts_tablet_manager.cc:1398] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:28.400084 31154 ts_tablet_manager.cc:1367] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20240501 14:01:28.400751 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:28.400815 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:28.401028 30032 raft_consensus.cc:2454] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:28.401077 31154 raft_consensus.cc:348] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.401326 31154 raft_consensus.cc:374] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.401505 31154 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
W20240501 14:01:28.401613 30228 leader_election.cc:341] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:28.401769 31154 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.401957 30228 leader_election.cc:302] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 76c7a691de8a4e31b32799f43cb4b812
I20240501 14:01:28.402915 31154 ts_tablet_manager.cc:1398] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:28.403883 31151 ts_tablet_manager.cc:1398] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:01:28.404502 30915 raft_consensus.cc:2781] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:28.404600 30915 raft_consensus.cc:482] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:28.404711 30915 raft_consensus.cc:3037] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.404795 31151 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:28.406414 31151 tablet_bootstrap.cc:654] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:28.407548 30915 raft_consensus.cc:504] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.408406 30915 leader_election.cc:288] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:28.408751 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:28.408834 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:28.409068 30032 raft_consensus.cc:3037] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:28.409250 30228 leader_election.cc:341] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:28.409699 31151 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:28.409843 31151 ts_tablet_manager.cc:1367] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:01:28.410573 31151 raft_consensus.cc:348] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.410766 31151 raft_consensus.cc:374] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:28.410861 31151 raft_consensus.cc:729] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:28.411082 31151 consensus_queue.cc:260] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.411608 31151 ts_tablet_manager.cc:1398] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:28.411787 30032 raft_consensus.cc:2454] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:28.412222 30228 leader_election.cc:302] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 76c7a691de8a4e31b32799f43cb4b812
I20240501 14:01:28.412493 30915 raft_consensus.cc:2781] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:28.412969 30915 raft_consensus.cc:686] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:28.413445 30915 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.416143 29754 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:28.442804 30895 raft_consensus.cc:482] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:28.443006 30895 raft_consensus.cc:504] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.443856 30895 leader_election.cc:288] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:28.444540 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:28.444540 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:28.444774 30160 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:28.444774 30032 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:28.445276 29837 leader_election.cc:302] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:28.445588 30895 raft_consensus.cc:2781] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:28.445698 30895 raft_consensus.cc:482] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:28.445772 30895 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.448113 30895 raft_consensus.cc:504] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.448737 30895 leader_election.cc:288] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:28.449292 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:28.449327 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:28.449512 30160 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.449496 30032 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.451501 30160 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:28.451553 30032 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:28.452031 29837 leader_election.cc:302] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:28.452459 30895 raft_consensus.cc:2781] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:28.452577 30895 raft_consensus.cc:686] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:28.452832 30895 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:28.455659 29754 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:28.532768 30895 raft_consensus.cc:482] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:28.532990 30895 raft_consensus.cc:504] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.533628 30895 leader_election.cc:288] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:28.534092 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c953ab40905a4d23b9b9c970d25168b9" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:28.534184 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c953ab40905a4d23b9b9c970d25168b9" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:28.534303 30160 raft_consensus.cc:2454] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:28.534451 30292 raft_consensus.cc:2454] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:28.534713 29837 leader_election.cc:302] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:28.535012 30895 raft_consensus.cc:2781] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:28.535118 30895 raft_consensus.cc:482] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:28.535238 30895 raft_consensus.cc:3037] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.537104 30895 raft_consensus.cc:504] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.537734 30895 leader_election.cc:288] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:28.538087 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c953ab40905a4d23b9b9c970d25168b9" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:28.538149 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c953ab40905a4d23b9b9c970d25168b9" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:28.538339 30292 raft_consensus.cc:3037] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.538341 30160 raft_consensus.cc:3037] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.541136 30292 raft_consensus.cc:2454] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:28.541134 30160 raft_consensus.cc:2454] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:28.541541 29836 leader_election.cc:302] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:28.541798 30895 raft_consensus.cc:2781] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:28.541975 30895 raft_consensus.cc:686] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:28.542237 30895 consensus_queue.cc:237] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.544960 29754 catalog_manager.cc:5484] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:28.793334 30919 raft_consensus.cc:482] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:28.793558 30919 raft_consensus.cc:504] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:28.794319 30919 leader_election.cc:288] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:28.794862 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:28.794878 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56df9dcda34e6a8d27a8a6b24742b2" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:28.795176 30032 raft_consensus.cc:2379] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c7a691de8a4e31b32799f43cb4b812 in current term 1: Already voted for candidate 4a15e881772b41eab8a71fc027390db9 in this term.
I20240501 14:01:28.795593 30098 leader_election.cc:302] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812; no voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:28.795850 30919 raft_consensus.cc:3037] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:28.798725 30919 raft_consensus.cc:2726] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:28.839308 30915 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:28.844657 30915 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:28.936623 31167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:28.937031 31167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:28.936964 30895 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:28.947054 31167 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:28.947053 30895 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 14:01:28.981781 31096 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:28.986899 30895 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.595 ms
time per row: 0.107798 ms
I20240501 14:01:29.324199 29752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:29.333226 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 0929994d486d4610b9cc76abd39f043a (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.333712 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929994d486d4610b9cc76abd39f043a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.333623 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 0929994d486d4610b9cc76abd39f043a (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.333714 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1ada0fe1ce431a823f2814f116db28 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.334043 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929994d486d4610b9cc76abd39f043a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.334157 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1ada0fe1ce431a823f2814f116db28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.334647 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 48b5a3abfd8348b9ae47c8867642044e (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.334962 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b5a3abfd8348b9ae47c8867642044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.338444 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 0929994d486d4610b9cc76abd39f043a (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.338857 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929994d486d4610b9cc76abd39f043a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.341758 31231 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:29.341742 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 48b5a3abfd8348b9ae47c8867642044e (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.342219 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b5a3abfd8348b9ae47c8867642044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.342370 31230 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:29.343148 29881 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1ada0fe1ce431a823f2814f116db28 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.343519 31231 tablet_bootstrap.cc:654] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.345562 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 48b5a3abfd8348b9ae47c8867642044e (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.346001 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b5a3abfd8348b9ae47c8867642044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.346962 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1ada0fe1ce431a823f2814f116db28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.355301 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1ada0fe1ce431a823f2814f116db28 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:29.357386 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1ada0fe1ce431a823f2814f116db28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:29.358682 31233 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:29.359252 31234 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:29.360347 31230 tablet_bootstrap.cc:654] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.361248 31233 tablet_bootstrap.cc:654] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.363871 31234 tablet_bootstrap.cc:654] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.368551 31231 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:29.368749 31231 ts_tablet_manager.cc:1367] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.008s
I20240501 14:01:29.368849 31233 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:29.368996 31233 ts_tablet_manager.cc:1367] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 14:01:29.369251 31230 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:29.369283 31234 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:29.369395 31230 ts_tablet_manager.cc:1367] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240501 14:01:29.369601 31231 raft_consensus.cc:348] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.369824 31231 raft_consensus.cc:374] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.369917 31231 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.369829 31233 raft_consensus.cc:348] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.370021 31233 raft_consensus.cc:374] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.370232 31233 raft_consensus.cc:729] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.370122 31230 raft_consensus.cc:348] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.370155 31231 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.370332 31230 raft_consensus.cc:374] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.370584 31230 raft_consensus.cc:729] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.370502 31233 consensus_queue.cc:260] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.370831 31230 consensus_queue.cc:260] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.371153 31234 ts_tablet_manager.cc:1367] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:01:29.371590 31233 ts_tablet_manager.cc:1398] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:29.372097 31230 ts_tablet_manager.cc:1398] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:29.372413 31233 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:29.372493 31230 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:29.372763 31231 ts_tablet_manager.cc:1398] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:29.373143 31231 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:29.374138 31233 tablet_bootstrap.cc:654] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.374138 31230 tablet_bootstrap.cc:654] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.375533 31231 tablet_bootstrap.cc:654] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.382581 31231 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:29.382897 31231 ts_tablet_manager.cc:1367] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240501 14:01:29.383469 31234 raft_consensus.cc:348] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.383821 31234 raft_consensus.cc:374] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.383777 31231 raft_consensus.cc:348] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.384022 31231 raft_consensus.cc:374] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.383956 31234 raft_consensus.cc:729] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.384102 31231 raft_consensus.cc:729] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.384379 31231 consensus_queue.cc:260] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.384436 31234 consensus_queue.cc:260] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.385288 31234 ts_tablet_manager.cc:1398] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:29.387073 31231 ts_tablet_manager.cc:1398] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:01:29.388124 31234 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:29.389593 31233 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:29.389851 31233 ts_tablet_manager.cc:1367] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.012s
I20240501 14:01:29.390995 31234 tablet_bootstrap.cc:654] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.390800 31233 raft_consensus.cc:348] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.391156 31233 raft_consensus.cc:374] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.391316 31233 raft_consensus.cc:729] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.391661 31233 consensus_queue.cc:260] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.392233 31233 ts_tablet_manager.cc:1398] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:29.392669 31233 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:29.394263 31233 tablet_bootstrap.cc:654] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:29.394590 31230 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:29.394656 31234 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:29.394856 31230 ts_tablet_manager.cc:1367] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20240501 14:01:29.394923 31234 ts_tablet_manager.cc:1367] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:01:29.396729 31230 raft_consensus.cc:348] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.396729 31234 raft_consensus.cc:348] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.397282 31234 raft_consensus.cc:374] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.397475 31234 raft_consensus.cc:729] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.398053 31234 consensus_queue.cc:260] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.397310 31230 raft_consensus.cc:374] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.399375 31230 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.400108 31230 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.400676 31234 ts_tablet_manager.cc:1398] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:01:29.401572 31230 ts_tablet_manager.cc:1398] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:01:29.403542 31233 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:29.403697 31233 ts_tablet_manager.cc:1367] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 14:01:29.404399 31233 raft_consensus.cc:348] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.404624 31233 raft_consensus.cc:374] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:29.404722 31233 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:29.405037 31233 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.405625 31233 ts_tablet_manager.cc:1398] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:29.564249 31037 raft_consensus.cc:482] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:29.564428 31037 raft_consensus.cc:504] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.565063 31037 leader_election.cc:288] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:29.565543 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0929994d486d4610b9cc76abd39f043a" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:29.565586 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0929994d486d4610b9cc76abd39f043a" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:29.565750 29901 raft_consensus.cc:2454] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:29.565786 30160 raft_consensus.cc:2454] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:29.566169 29965 leader_election.cc:302] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:29.566463 31037 raft_consensus.cc:2781] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:29.566581 31037 raft_consensus.cc:482] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:29.566673 31037 raft_consensus.cc:3037] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.568507 31037 raft_consensus.cc:504] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.569139 31037 leader_election.cc:288] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:29.569497 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0929994d486d4610b9cc76abd39f043a" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:29.569707 29901 raft_consensus.cc:3037] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.569650 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0929994d486d4610b9cc76abd39f043a" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:29.569836 30160 raft_consensus.cc:3037] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.572639 30160 raft_consensus.cc:2454] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:29.572628 29901 raft_consensus.cc:2454] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:29.573233 29968 leader_election.cc:302] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:29.573523 31243 raft_consensus.cc:2781] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:29.573684 31243 raft_consensus.cc:686] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:29.573992 31243 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:29.576786 29752 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:29.630107 31243 raft_consensus.cc:482] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:29.630337 31243 raft_consensus.cc:504] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.631121 31243 leader_election.cc:288] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:29.631733 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:29.631995 29901 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:29.632454 29965 leader_election.cc:302] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:29.632784 31243 raft_consensus.cc:2781] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:29.632889 31243 raft_consensus.cc:482] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:29.633015 31243 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.635794 31243 raft_consensus.cc:504] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.636525 31243 leader_election.cc:288] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:29.636845 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:29.637081 29901 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.639688 29901 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:29.640136 29965 leader_election.cc:302] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:29.640444 31243 raft_consensus.cc:2781] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:29.640620 31243 raft_consensus.cc:686] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:29.640864 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:29.640921 31243 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:29.640888 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:29.641173 30292 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:29.643980 29752 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:29.720710 30919 raft_consensus.cc:482] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:29.720945 30919 raft_consensus.cc:504] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.721705 30919 leader_election.cc:288] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:29.722215 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b5a3abfd8348b9ae47c8867642044e" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:29.722245 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b5a3abfd8348b9ae47c8867642044e" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:29.722446 30290 raft_consensus.cc:2454] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:29.722496 29901 raft_consensus.cc:2454] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:29.722896 30097 leader_election.cc:302] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:29.723213 30919 raft_consensus.cc:2781] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:29.723299 30919 raft_consensus.cc:482] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:29.723404 30919 raft_consensus.cc:3037] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.725762 30919 raft_consensus.cc:504] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.726460 30919 leader_election.cc:288] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:29.726847 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b5a3abfd8348b9ae47c8867642044e" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:29.726996 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b5a3abfd8348b9ae47c8867642044e" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:29.727079 29901 raft_consensus.cc:3037] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.727231 30290 raft_consensus.cc:3037] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:29.729151 29901 raft_consensus.cc:2454] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:29.729115 30290 raft_consensus.cc:2454] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:29.729635 30095 leader_election.cc:302] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:29.729931 30919 raft_consensus.cc:2781] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:29.730104 30919 raft_consensus.cc:686] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:29.730373 30919 consensus_queue.cc:237] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:29.733224 29752 catalog_manager.cc:5484] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:30.028724 31243 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:30.036583 31037 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:30.113009 30919 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:30.118114 31257 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:30.152070 30290 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.156517 31037 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:30.161226 31243 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:30.197345 31250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:30.197750 31250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:30.205541 31250 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 214.664 ms
time per row: 0.107332 ms
I20240501 14:01:30.590281 29759 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:30.600036 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 6241869f8e7f4bfaa65508fd61f405a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.600306 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 6241869f8e7f4bfaa65508fd61f405a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.600570 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6241869f8e7f4bfaa65508fd61f405a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.600759 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6241869f8e7f4bfaa65508fd61f405a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.601079 29880 tablet_service.cc:1467] Processing CreateTablet for tablet c00649352f9d41c3a12a5e767b8b2aa6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.601621 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00649352f9d41c3a12a5e767b8b2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.602366 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 6241869f8e7f4bfaa65508fd61f405a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.602550 30272 tablet_service.cc:1467] Processing CreateTablet for tablet f4988f0de3214860a65007b8e31e88a5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.602754 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6241869f8e7f4bfaa65508fd61f405a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.602949 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4988f0de3214860a65007b8e31e88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.605517 30140 tablet_service.cc:1467] Processing CreateTablet for tablet f4988f0de3214860a65007b8e31e88a5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.605957 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4988f0de3214860a65007b8e31e88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.609227 30142 tablet_service.cc:1467] Processing CreateTablet for tablet c00649352f9d41c3a12a5e767b8b2aa6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.609752 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00649352f9d41c3a12a5e767b8b2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.614470 30011 tablet_service.cc:1467] Processing CreateTablet for tablet f4988f0de3214860a65007b8e31e88a5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.615042 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4988f0de3214860a65007b8e31e88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.617554 31309 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:30.617516 30012 tablet_service.cc:1467] Processing CreateTablet for tablet c00649352f9d41c3a12a5e767b8b2aa6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:30.617864 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00649352f9d41c3a12a5e767b8b2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:30.619259 31309 tablet_bootstrap.cc:654] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.632532 31310 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:30.634929 31310 tablet_bootstrap.cc:654] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.640302 31309 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:30.640553 31309 ts_tablet_manager.cc:1367] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.007s
I20240501 14:01:30.641484 31309 raft_consensus.cc:348] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.641885 31309 raft_consensus.cc:374] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.642027 31309 raft_consensus.cc:729] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.642355 31309 consensus_queue.cc:260] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.643101 31309 ts_tablet_manager.cc:1398] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:30.643818 31309 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:30.645872 31309 tablet_bootstrap.cc:654] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.654354 31310 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:30.654558 31310 ts_tablet_manager.cc:1367] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 14:01:30.655524 31310 raft_consensus.cc:348] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.655762 31310 raft_consensus.cc:374] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.655854 31310 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.655691 31309 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:30.656204 31310 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.656363 31309 ts_tablet_manager.cc:1367] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240501 14:01:30.656831 31310 ts_tablet_manager.cc:1398] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:30.658125 31309 raft_consensus.cc:348] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.658766 31309 raft_consensus.cc:374] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.658841 31309 raft_consensus.cc:729] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.659073 31309 consensus_queue.cc:260] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.660199 31309 ts_tablet_manager.cc:1398] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:30.668241 31315 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:30.669497 31311 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:30.670168 31315 tablet_bootstrap.cc:654] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.671026 31311 tablet_bootstrap.cc:654] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.672355 31315 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:30.672686 31315 ts_tablet_manager.cc:1367] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:01:30.674835 31315 raft_consensus.cc:348] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.675217 31315 raft_consensus.cc:374] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.675392 31315 raft_consensus.cc:729] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.675791 31315 consensus_queue.cc:260] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.676684 31315 ts_tablet_manager.cc:1398] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:30.677099 31315 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:30.679131 31315 tablet_bootstrap.cc:654] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.680714 31311 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:30.680898 31311 ts_tablet_manager.cc:1367] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:01:30.682010 31311 raft_consensus.cc:348] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.682228 31311 raft_consensus.cc:374] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.682323 31311 raft_consensus.cc:729] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.682639 31311 consensus_queue.cc:260] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.684368 31311 ts_tablet_manager.cc:1398] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:30.684734 31311 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:30.686779 31248 raft_consensus.cc:482] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:30.686980 31248 raft_consensus.cc:504] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.687928 31248 leader_election.cc:288] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:30.688720 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:30.688931 30290 raft_consensus.cc:2454] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:30.689524 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:30.689952 31311 tablet_bootstrap.cc:654] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
W20240501 14:01:30.690347 30098 leader_election.cc:341] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:30.690749 30097 leader_election.cc:302] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters: 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:30.691236 31248 raft_consensus.cc:2781] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:30.691337 31248 raft_consensus.cc:482] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:30.691403 31248 raft_consensus.cc:3037] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.694427 31248 raft_consensus.cc:504] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.695013 31248 leader_election.cc:288] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:30.695849 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
W20240501 14:01:30.696571 30098 leader_election.cc:341] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:30.696622 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:30.696826 30290 raft_consensus.cc:3037] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.697484 31315 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:30.697682 31311 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:30.697698 31315 ts_tablet_manager.cc:1367] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20240501 14:01:30.697822 31311 ts_tablet_manager.cc:1367] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240501 14:01:30.698503 31311 raft_consensus.cc:348] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.698704 31311 raft_consensus.cc:374] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.698580 31315 raft_consensus.cc:348] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.698776 31311 raft_consensus.cc:729] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.698856 31315 raft_consensus.cc:374] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.698931 31315 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.699051 31311 consensus_queue.cc:260] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.699188 31315 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.699661 30290 raft_consensus.cc:2454] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:30.700244 31315 ts_tablet_manager.cc:1398] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:30.700304 31311 ts_tablet_manager.cc:1398] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:30.700544 31311 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:30.700556 31315 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:30.702215 31311 tablet_bootstrap.cc:654] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.702215 31315 tablet_bootstrap.cc:654] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:30.702944 30097 leader_election.cc:302] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters: 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:30.707962 31248 raft_consensus.cc:2781] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:30.708180 31248 raft_consensus.cc:686] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:30.708480 31248 consensus_queue.cc:237] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.713287 31311 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:30.713325 31315 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:30.713423 31311 ts_tablet_manager.cc:1367] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240501 14:01:30.713495 31315 ts_tablet_manager.cc:1367] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240501 14:01:30.713986 31311 raft_consensus.cc:348] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.713790 29759 catalog_manager.cc:5484] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:30.714134 31311 raft_consensus.cc:374] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.714195 31311 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.714349 31311 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.714319 31315 raft_consensus.cc:348] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.714529 31315 raft_consensus.cc:374] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:30.714617 31315 raft_consensus.cc:729] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:30.714707 31311 ts_tablet_manager.cc:1398] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:30.714859 31315 consensus_queue.cc:260] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:30.715327 31315 ts_tablet_manager.cc:1398] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:30.739703 31261 raft_consensus.cc:482] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:30.739945 31261 raft_consensus.cc:504] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.740743 31261 leader_election.cc:288] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:30.741261 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:30.741367 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:30.741473 29900 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:30.741647 30160 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:30.741957 29965 leader_election.cc:302] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:30.742269 31261 raft_consensus.cc:2781] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:30.742400 31261 raft_consensus.cc:482] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:30.742506 31261 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.744410 31261 raft_consensus.cc:504] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.745039 31261 leader_election.cc:288] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:30.745558 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:30.745580 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:30.745754 29900 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.745790 30160 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.747692 29900 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:30.747728 30160 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:30.748129 29965 leader_election.cc:302] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:30.748487 31261 raft_consensus.cc:2781] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:30.748648 31261 raft_consensus.cc:686] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:30.748934 31261 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:30.752002 29759 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:30.827054 31261 raft_consensus.cc:482] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:30.827271 31261 raft_consensus.cc:504] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.827883 31261 leader_election.cc:288] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:30.828410 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4988f0de3214860a65007b8e31e88a5" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:30.828450 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4988f0de3214860a65007b8e31e88a5" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:30.828615 30160 raft_consensus.cc:2454] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:30.828703 30290 raft_consensus.cc:2454] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:30.829030 29968 leader_election.cc:302] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:30.829311 31261 raft_consensus.cc:2781] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:30.829416 31261 raft_consensus.cc:482] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:30.829499 31261 raft_consensus.cc:3037] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.831359 31261 raft_consensus.cc:504] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.831928 31261 leader_election.cc:288] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:30.832332 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4988f0de3214860a65007b8e31e88a5" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:30.832471 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4988f0de3214860a65007b8e31e88a5" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:30.832553 30290 raft_consensus.cc:3037] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.832671 30160 raft_consensus.cc:3037] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:30.834626 30290 raft_consensus.cc:2454] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:30.834632 30160 raft_consensus.cc:2454] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:30.835054 29967 leader_election.cc:302] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:30.835402 31261 raft_consensus.cc:2781] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:30.835578 31261 raft_consensus.cc:686] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:30.835819 31261 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:30.838882 29759 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:31.143013 31261 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:31.144784 31261 raft_consensus.cc:482] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:31.144979 31261 raft_consensus.cc:504] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:31.145680 31261 leader_election.cc:288] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:31.148298 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:31.148582 30290 raft_consensus.cc:2379] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in current term 1: Already voted for candidate 76c7a691de8a4e31b32799f43cb4b812 in this term.
I20240501 14:01:31.149284 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6241869f8e7f4bfaa65508fd61f405a6" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:31.154791 30032 raft_consensus.cc:3037] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.160648 31248 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:31.161216 29968 leader_election.cc:302] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812
I20240501 14:01:31.162097 31328 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:31.173898 31261 raft_consensus.cc:2726] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:31.182933 31248 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:31.250979 31261 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:31.253440 31325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:31.253921 31325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:31.255721 31261 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:31.264797 31325 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 225.856 ms
time per row: 0.112928 ms
I20240501 14:01:31.654188 29752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:31.663928 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 31303ad8c9124217a45e26f8da5cc0a1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.664203 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc0201075e1466a9661b155c180d95f (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.664283 29881 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc0201075e1466a9661b155c180d95f (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.664492 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31303ad8c9124217a45e26f8da5cc0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.664543 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 31303ad8c9124217a45e26f8da5cc0a1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.664736 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc0201075e1466a9661b155c180d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.664881 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 31303ad8c9124217a45e26f8da5cc0a1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.665328 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31303ad8c9124217a45e26f8da5cc0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.665339 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31303ad8c9124217a45e26f8da5cc0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.669688 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc0201075e1466a9661b155c180d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.669672 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc0201075e1466a9661b155c180d95f (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.670126 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc0201075e1466a9661b155c180d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.677691 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.678210 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.681679 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.682133 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.685787 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:31.686390 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:31.687186 31397 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:31.691246 31396 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:31.692811 31397 tablet_bootstrap.cc:654] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.695744 31396 tablet_bootstrap.cc:654] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.701495 31396 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:31.701900 31396 ts_tablet_manager.cc:1367] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240501 14:01:31.703042 31396 raft_consensus.cc:348] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.704391 31396 raft_consensus.cc:374] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.704742 31396 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.705170 31396 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.706606 31396 ts_tablet_manager.cc:1398] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:31.707710 31396 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:31.712803 31396 tablet_bootstrap.cc:654] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.721819 31396 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:31.722929 31396 ts_tablet_manager.cc:1367] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240501 14:01:31.724426 31396 raft_consensus.cc:348] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.725848 31395 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:31.726289 31396 raft_consensus.cc:374] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.726516 31398 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:31.726629 31396 raft_consensus.cc:729] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.727974 31396 consensus_queue.cc:260] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.727880 31397 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:31.728770 31396 ts_tablet_manager.cc:1398] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:31.729103 31398 tablet_bootstrap.cc:654] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.729169 31396 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:31.729523 31397 ts_tablet_manager.cc:1367] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.015s
I20240501 14:01:31.730451 31397 raft_consensus.cc:348] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.730715 31397 raft_consensus.cc:374] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.730839 31397 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.730857 31396 tablet_bootstrap.cc:654] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.731117 31397 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.731820 31395 tablet_bootstrap.cc:654] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.732878 31397 ts_tablet_manager.cc:1398] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:01:31.733178 31397 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:31.734892 31397 tablet_bootstrap.cc:654] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.738934 31396 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:31.739121 31396 ts_tablet_manager.cc:1367] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 14:01:31.739995 31396 raft_consensus.cc:348] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.740243 31396 raft_consensus.cc:374] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.740347 31396 raft_consensus.cc:729] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.740624 31396 consensus_queue.cc:260] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.741336 31396 ts_tablet_manager.cc:1398] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:31.745039 31397 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:31.745296 31397 ts_tablet_manager.cc:1367] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240501 14:01:31.746290 31397 raft_consensus.cc:348] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.746577 31395 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:31.746560 31397 raft_consensus.cc:374] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.746663 31397 raft_consensus.cc:729] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.746744 31395 ts_tablet_manager.cc:1367] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20240501 14:01:31.746941 31397 consensus_queue.cc:260] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.747591 31395 raft_consensus.cc:348] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.747802 31395 raft_consensus.cc:374] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.747903 31395 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.748131 31395 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.748368 31397 ts_tablet_manager.cc:1398] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:31.748819 31395 ts_tablet_manager.cc:1398] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:31.749079 31395 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:31.750417 31395 tablet_bootstrap.cc:654] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.752041 31398 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:31.752215 31398 ts_tablet_manager.cc:1367] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.000s
I20240501 14:01:31.753026 31398 raft_consensus.cc:348] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.753358 31398 raft_consensus.cc:374] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.753502 31398 raft_consensus.cc:729] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.753767 31398 consensus_queue.cc:260] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.754448 31398 ts_tablet_manager.cc:1398] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:31.755728 31395 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:31.755908 31395 ts_tablet_manager.cc:1367] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:01:31.756739 31395 raft_consensus.cc:348] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.756966 31395 raft_consensus.cc:374] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.757036 31395 raft_consensus.cc:729] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.757323 31395 consensus_queue.cc:260] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.758829 31395 ts_tablet_manager.cc:1398] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:31.759131 31395 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:31.761641 31395 tablet_bootstrap.cc:654] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:31.763593 31395 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:31.763737 31395 ts_tablet_manager.cc:1367] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:01:31.764473 31395 raft_consensus.cc:348] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.764689 31395 raft_consensus.cc:374] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:31.764772 31395 raft_consensus.cc:729] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:31.765000 31395 consensus_queue.cc:260] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.765578 31395 ts_tablet_manager.cc:1398] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:31.782986 30919 raft_consensus.cc:482] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:31.783198 30919 raft_consensus.cc:504] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.783825 30919 leader_election.cc:288] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:31.784353 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc0201075e1466a9661b155c180d95f" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:31.784409 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc0201075e1466a9661b155c180d95f" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:31.784581 29900 raft_consensus.cc:2454] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:31.784660 30290 raft_consensus.cc:2454] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:31.785043 30095 leader_election.cc:302] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:31.785367 30919 raft_consensus.cc:2781] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:31.785475 30919 raft_consensus.cc:482] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:31.785571 30919 raft_consensus.cc:3037] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.787478 30919 raft_consensus.cc:504] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.788094 30919 leader_election.cc:288] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:31.788477 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc0201075e1466a9661b155c180d95f" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:31.788583 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc0201075e1466a9661b155c180d95f" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:31.788681 30290 raft_consensus.cc:3037] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.788815 29900 raft_consensus.cc:3037] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.790825 30290 raft_consensus.cc:2454] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:31.790844 29900 raft_consensus.cc:2454] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 1.
I20240501 14:01:31.791321 30095 leader_election.cc:302] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:31.791653 30919 raft_consensus.cc:2781] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:31.791918 30919 raft_consensus.cc:686] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:31.792325 30919 consensus_queue.cc:237] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:31.796018 29752 catalog_manager.cc:5484] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:31.869333 31178 raft_consensus.cc:482] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:31.869587 31178 raft_consensus.cc:504] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.869868 31337 raft_consensus.cc:482] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:31.870023 31337 raft_consensus.cc:504] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.870412 31178 leader_election.cc:288] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:31.870855 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:31.870968 31337 leader_election.cc:288] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:31.870925 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:31.871126 30290 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:31.871234 30031 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:31.871778 29837 leader_election.cc:302] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:31.872023 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:31.872121 31178 raft_consensus.cc:2781] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:31.872257 31178 raft_consensus.cc:482] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:31.872259 30031 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:31.872375 31178 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.872742 30228 leader_election.cc:302] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:31.873045 31337 raft_consensus.cc:2781] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:31.873196 31337 raft_consensus.cc:482] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:31.873291 31337 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.875108 31178 raft_consensus.cc:504] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.875777 31178 leader_election.cc:288] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:31.876214 31337 raft_consensus.cc:504] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.876266 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:31.876439 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:31.876621 30031 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:31.876678 30290 raft_consensus.cc:2379] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7869901e77234aeeb1462876b63eb28e in current term 1: Already voted for candidate 4a15e881772b41eab8a71fc027390db9 in this term.
I20240501 14:01:31.876940 31337 leader_election.cc:288] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:31.877774 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:31.879288 30031 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:31.879689 29837 leader_election.cc:302] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 4a15e881772b41eab8a71fc027390db9
I20240501 14:01:31.879753 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:31.879802 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:31.880051 29900 raft_consensus.cc:2379] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a15e881772b41eab8a71fc027390db9 in current term 1: Already voted for candidate 7869901e77234aeeb1462876b63eb28e in this term.
I20240501 14:01:31.880211 31178 raft_consensus.cc:2781] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:31.880342 30225 leader_election.cc:302] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9; no voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:31.880698 31178 raft_consensus.cc:686] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:31.880719 31337 raft_consensus.cc:2726] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:01:31.881026 31178 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:31.884246 29752 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:32.028487 31178 raft_consensus.cc:482] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:32.028752 31178 raft_consensus.cc:504] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:32.029497 31178 leader_election.cc:288] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:32.030222 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea9aaebd437424eb27c04bc2f6ffa7b" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:32.030222 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea9aaebd437424eb27c04bc2f6ffa7b" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:32.030508 30031 raft_consensus.cc:2454] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:32.030542 30290 raft_consensus.cc:2454] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:32.031045 29836 leader_election.cc:302] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:32.031385 31178 raft_consensus.cc:2781] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:32.031474 31178 raft_consensus.cc:482] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:32.031553 31178 raft_consensus.cc:3037] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.033689 31178 raft_consensus.cc:504] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:32.034442 31178 leader_election.cc:288] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:32.035059 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea9aaebd437424eb27c04bc2f6ffa7b" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:32.035143 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea9aaebd437424eb27c04bc2f6ffa7b" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:32.035290 30290 raft_consensus.cc:3037] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.035393 30031 raft_consensus.cc:3037] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.037945 30290 raft_consensus.cc:2454] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:32.038396 29836 leader_election.cc:302] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:32.038731 31178 raft_consensus.cc:2781] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:32.038933 31178 raft_consensus.cc:686] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:32.038940 30031 raft_consensus.cc:2454] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:32.039263 31178 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:32.042511 29752 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:32.253378 31248 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.268114 30919 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.382143 31178 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.391353 31411 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.445406 31424 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.450743 31422 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:32.530900 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:32.531388 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:32.539316 31414 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240501 14:01:32.819427 29965 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:35665, user_credentials={real_user=slave}} blocked reactor thread for 71500.5us
INSERT report
rows total: 2000
time total: 287.415 ms
time per row: 0.143707 ms
I20240501 14:01:32.976210 29752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:32.985339 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 368edc181c894686a213fc542f328192 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.985854 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368edc181c894686a213fc542f328192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.985810 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 368edc181c894686a213fc542f328192 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.986184 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368edc181c894686a213fc542f328192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.986285 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 56f632889b1e400392fe710bf9f6f3ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.986737 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f632889b1e400392fe710bf9f6f3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.989941 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 56f632889b1e400392fe710bf9f6f3ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.990360 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f632889b1e400392fe710bf9f6f3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.992156 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 368edc181c894686a213fc542f328192 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.992730 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368edc181c894686a213fc542f328192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.997608 30270 tablet_service.cc:1467] Processing CreateTablet for tablet edf4bcf02c7e4352bd998e139802db62 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.998237 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf4bcf02c7e4352bd998e139802db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.998157 30142 tablet_service.cc:1467] Processing CreateTablet for tablet 56f632889b1e400392fe710bf9f6f3ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.998481 31479 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:32.998616 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f632889b1e400392fe710bf9f6f3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.999341 31478 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:33.000349 31479 tablet_bootstrap.cc:654] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.001174 31478 tablet_bootstrap.cc:654] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.001590 30140 tablet_service.cc:1467] Processing CreateTablet for tablet edf4bcf02c7e4352bd998e139802db62 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.001976 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf4bcf02c7e4352bd998e139802db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.001927 30011 tablet_service.cc:1467] Processing CreateTablet for tablet edf4bcf02c7e4352bd998e139802db62 (DEFAULT_TABLE table=rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.002449 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf4bcf02c7e4352bd998e139802db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.021762 31479 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:33.022112 31479 ts_tablet_manager.cc:1367] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20240501 14:01:33.023216 31479 raft_consensus.cc:348] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.023687 31479 raft_consensus.cc:374] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.023841 31479 raft_consensus.cc:729] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.024194 31479 consensus_queue.cc:260] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.024888 31479 ts_tablet_manager.cc:1398] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:33.031569 31479 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:33.035665 31482 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:33.038278 31482 tablet_bootstrap.cc:654] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.040122 31479 tablet_bootstrap.cc:654] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.042430 31483 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:33.042740 31479 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:33.042941 31479 ts_tablet_manager.cc:1367] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:01:33.043351 31478 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:33.043531 31478 ts_tablet_manager.cc:1367] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.021s
I20240501 14:01:33.043875 31479 raft_consensus.cc:348] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.044165 31479 raft_consensus.cc:374] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.044289 31479 raft_consensus.cc:729] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.044430 31482 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:33.044386 31478 raft_consensus.cc:348] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.044591 31478 raft_consensus.cc:374] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.044730 31482 ts_tablet_manager.cc:1367] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:01:33.044607 31479 consensus_queue.cc:260] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.044749 31478 raft_consensus.cc:729] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.044858 31483 tablet_bootstrap.cc:654] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.045085 31478 consensus_queue.cc:260] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.045274 31479 ts_tablet_manager.cc:1398] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.045914 31478 ts_tablet_manager.cc:1398] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:33.046248 31478 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:33.048092 31478 tablet_bootstrap.cc:654] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.053236 31483 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:33.053436 31483 ts_tablet_manager.cc:1367] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 14:01:33.054332 31483 raft_consensus.cc:348] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.054571 31483 raft_consensus.cc:374] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.054714 31483 raft_consensus.cc:729] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.055001 31483 consensus_queue.cc:260] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.056558 31482 raft_consensus.cc:348] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.056803 31482 raft_consensus.cc:374] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.057104 31482 raft_consensus.cc:729] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.057219 31483 ts_tablet_manager.cc:1398] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:33.057554 31483 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:33.059422 31483 tablet_bootstrap.cc:654] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.060792 31482 consensus_queue.cc:260] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.062800 31483 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:33.063040 31483 ts_tablet_manager.cc:1367] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:33.063896 31483 raft_consensus.cc:348] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.064217 31483 raft_consensus.cc:374] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.064314 31483 raft_consensus.cc:729] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.064613 31483 consensus_queue.cc:260] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.065428 31483 ts_tablet_manager.cc:1398] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:33.061520 31482 ts_tablet_manager.cc:1398] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:33.066412 31482 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:33.068500 31478 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:33.068778 31478 ts_tablet_manager.cc:1367] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.016s
I20240501 14:01:33.069183 31482 tablet_bootstrap.cc:654] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.069943 31478 raft_consensus.cc:348] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.070152 31478 raft_consensus.cc:374] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.070240 31478 raft_consensus.cc:729] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.070511 31478 consensus_queue.cc:260] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.071087 31478 ts_tablet_manager.cc:1398] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.071250 31482 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:33.071453 31482 ts_tablet_manager.cc:1367] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:01:33.072244 31482 raft_consensus.cc:348] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.072512 31482 raft_consensus.cc:374] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.072599 31482 raft_consensus.cc:729] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.072853 31482 consensus_queue.cc:260] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.073516 31482 ts_tablet_manager.cc:1398] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:01:33.073866 31482 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:33.075521 31482 tablet_bootstrap.cc:654] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.076385 31411 raft_consensus.cc:482] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.076573 31411 raft_consensus.cc:504] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.077381 31411 leader_election.cc:288] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:33.078155 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:33.078146 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:33.078361 30031 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:33.078363 30162 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:33.078850 29837 leader_election.cc:302] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:33.079147 31482 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:33.079303 31482 ts_tablet_manager.cc:1367] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:33.079383 31411 raft_consensus.cc:2781] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.079478 31411 raft_consensus.cc:482] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.079597 31411 raft_consensus.cc:3037] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.080554 31482 raft_consensus.cc:348] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.080796 31482 raft_consensus.cc:374] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.080899 31482 raft_consensus.cc:729] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.081169 31482 consensus_queue.cc:260] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.081796 31482 ts_tablet_manager.cc:1398] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:33.082458 31411 raft_consensus.cc:504] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.083181 31411 leader_election.cc:288] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:33.083602 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:33.083667 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:33.083813 30162 raft_consensus.cc:3037] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.083881 30031 raft_consensus.cc:3037] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.086264 30162 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:33.086347 30031 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:33.086738 29837 leader_election.cc:302] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:33.087067 31411 raft_consensus.cc:2781] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.087304 31411 raft_consensus.cc:686] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:33.087620 31411 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.090797 29753 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:33.196081 31337 raft_consensus.cc:482] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.196346 31337 raft_consensus.cc:504] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.197069 31337 leader_election.cc:288] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:33.197717 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:33.197723 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:33.198002 30162 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:33.198002 30031 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:33.198460 30228 leader_election.cc:302] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:33.198791 31337 raft_consensus.cc:2781] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.198890 31337 raft_consensus.cc:482] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.198983 31337 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.201864 31337 raft_consensus.cc:504] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.202610 31337 leader_election.cc:288] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:33.203063 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:33.203218 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:33.203317 30162 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.203457 30031 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.206398 30031 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:33.206987 30228 leader_election.cc:302] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:33.207311 31337 raft_consensus.cc:2781] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.206398 30162 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:33.208878 31337 raft_consensus.cc:686] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:33.209206 31337 consensus_queue.cc:237] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:33.212992 29753 catalog_manager.cc:5484] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:33.225860 31337 raft_consensus.cc:482] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.226100 31337 raft_consensus.cc:504] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.226733 31337 leader_election.cc:288] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:33.227226 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f632889b1e400392fe710bf9f6f3ec" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:33.227245 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f632889b1e400392fe710bf9f6f3ec" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:33.227439 29900 raft_consensus.cc:2454] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:33.227501 30162 raft_consensus.cc:2454] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:33.227843 30225 leader_election.cc:302] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:33.228098 31337 raft_consensus.cc:2781] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.228220 31337 raft_consensus.cc:482] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.228303 31337 raft_consensus.cc:3037] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.230218 31337 raft_consensus.cc:504] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.230789 31337 leader_election.cc:288] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:33.231334 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f632889b1e400392fe710bf9f6f3ec" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:33.231544 30162 raft_consensus.cc:3037] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.233532 30162 raft_consensus.cc:2454] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:33.233892 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f632889b1e400392fe710bf9f6f3ec" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:33.234000 30228 leader_election.cc:302] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:33.234136 29900 raft_consensus.cc:3037] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.234366 31337 raft_consensus.cc:2781] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.234537 31337 raft_consensus.cc:686] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:33.234861 31337 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } }
I20240501 14:01:33.236773 29900 raft_consensus.cc:2454] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:33.238332 29753 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:33.511595 31411 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.517213 31411 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.635437 31337 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.646174 31337 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.660867 31492 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:33.707438 31495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:33.708227 31495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:33.709395 31337 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:33.722123 31495 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 262.751 ms
time per row: 0.131375 ms
I20240501 14:01:34.167676 29760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:34.177675 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 9735c5d9f784459ebafd8f2ff4f69763 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.177860 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 9735c5d9f784459ebafd8f2ff4f69763 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.178189 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9735c5d9f784459ebafd8f2ff4f69763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.178258 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9735c5d9f784459ebafd8f2ff4f69763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.180215 30141 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3626b1c8d458e9c6d4a28bc0564fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.180510 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3626b1c8d458e9c6d4a28bc0564fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.181641 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 594dcb0a05054f2abaa11e5d4537b724 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.182035 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594dcb0a05054f2abaa11e5d4537b724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.182658 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 9735c5d9f784459ebafd8f2ff4f69763 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.183079 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9735c5d9f784459ebafd8f2ff4f69763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.185595 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 594dcb0a05054f2abaa11e5d4537b724 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.185994 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594dcb0a05054f2abaa11e5d4537b724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.188948 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 594dcb0a05054f2abaa11e5d4537b724 (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.189144 29881 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3626b1c8d458e9c6d4a28bc0564fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.189425 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594dcb0a05054f2abaa11e5d4537b724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.189350 30272 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3626b1c8d458e9c6d4a28bc0564fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.189729 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3626b1c8d458e9c6d4a28bc0564fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.191663 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3626b1c8d458e9c6d4a28bc0564fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.204610 31556 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:34.207610 31556 tablet_bootstrap.cc:654] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.209614 31557 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:34.211874 31557 tablet_bootstrap.cc:654] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.213606 31554 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:34.217439 31557 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:34.217729 31557 ts_tablet_manager.cc:1367] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240501 14:01:34.218891 31557 raft_consensus.cc:348] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.219362 31557 raft_consensus.cc:374] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.219542 31557 raft_consensus.cc:729] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.219926 31557 consensus_queue.cc:260] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.220929 31554 tablet_bootstrap.cc:654] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.221222 31557 ts_tablet_manager.cc:1398] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:34.221669 31557 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:34.222523 31555 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:34.224197 31557 tablet_bootstrap.cc:654] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.227202 31555 tablet_bootstrap.cc:654] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.227654 31556 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:34.227792 31556 ts_tablet_manager.cc:1367] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.004s
I20240501 14:01:34.228508 31556 raft_consensus.cc:348] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.228740 31556 raft_consensus.cc:374] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.228845 31556 raft_consensus.cc:729] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.229080 31556 consensus_queue.cc:260] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.229710 31556 ts_tablet_manager.cc:1398] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.229996 31556 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:34.231649 31556 tablet_bootstrap.cc:654] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.233621 31557 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:34.233644 31556 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:34.233812 31557 ts_tablet_manager.cc:1367] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240501 14:01:34.233951 31556 ts_tablet_manager.cc:1367] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:34.234252 31555 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:34.234452 31555 ts_tablet_manager.cc:1367] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240501 14:01:34.234589 31557 raft_consensus.cc:348] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.234825 31557 raft_consensus.cc:374] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.234907 31557 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.234834 31556 raft_consensus.cc:348] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.235085 31556 raft_consensus.cc:374] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.235174 31556 raft_consensus.cc:729] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.235190 31557 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.235714 31555 raft_consensus.cc:348] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.235896 31557 ts_tablet_manager.cc:1398] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.235457 31556 consensus_queue.cc:260] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.236697 31555 raft_consensus.cc:374] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.236848 31555 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.236860 31556 ts_tablet_manager.cc:1398] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:34.239742 31554 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:34.239924 31554 ts_tablet_manager.cc:1367] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240501 14:01:34.240787 31554 raft_consensus.cc:348] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.240962 31554 raft_consensus.cc:374] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.241099 31554 raft_consensus.cc:729] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.241389 31554 consensus_queue.cc:260] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.241971 31554 ts_tablet_manager.cc:1398] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.237650 31555 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.242630 31555 ts_tablet_manager.cc:1398] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:01:34.243052 31554 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:34.244414 31555 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:34.244951 31554 tablet_bootstrap.cc:654] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.246284 31555 tablet_bootstrap.cc:654] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.255395 31555 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:34.255398 31554 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:34.255614 31554 ts_tablet_manager.cc:1367] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240501 14:01:34.255806 31555 ts_tablet_manager.cc:1367] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240501 14:01:34.256476 31554 raft_consensus.cc:348] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.256561 31555 raft_consensus.cc:348] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.256733 31554 raft_consensus.cc:374] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.256784 31555 raft_consensus.cc:374] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.256824 31554 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.256868 31555 raft_consensus.cc:729] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.257148 31555 consensus_queue.cc:260] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.257104 31554 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.257730 31555 ts_tablet_manager.cc:1398] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.257727 31554 ts_tablet_manager.cc:1398] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:34.258096 31554 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:34.259634 31554 tablet_bootstrap.cc:654] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.261618 31554 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:34.261737 31554 ts_tablet_manager.cc:1367] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:01:34.262302 31554 raft_consensus.cc:348] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.262455 31554 raft_consensus.cc:374] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.262540 31554 raft_consensus.cc:729] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.262740 31554 consensus_queue.cc:260] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.263159 31554 ts_tablet_manager.cc:1398] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:34.414870 31549 raft_consensus.cc:482] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.415076 31549 raft_consensus.cc:504] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.415889 31549 leader_election.cc:288] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:34.416432 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9735c5d9f784459ebafd8f2ff4f69763" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:34.416438 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9735c5d9f784459ebafd8f2ff4f69763" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:34.416704 29900 raft_consensus.cc:2454] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:34.416756 30162 raft_consensus.cc:2454] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:34.417249 29965 leader_election.cc:302] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:34.417580 31549 raft_consensus.cc:2781] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.417699 31549 raft_consensus.cc:482] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.417769 31549 raft_consensus.cc:3037] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.419734 31549 raft_consensus.cc:504] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.420341 31549 leader_election.cc:288] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:34.420855 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9735c5d9f784459ebafd8f2ff4f69763" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:34.420970 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9735c5d9f784459ebafd8f2ff4f69763" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:34.421141 29900 raft_consensus.cc:3037] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.421239 30162 raft_consensus.cc:3037] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.423128 30162 raft_consensus.cc:2454] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:34.423128 29900 raft_consensus.cc:2454] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:34.423537 29965 leader_election.cc:302] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:34.423867 31549 raft_consensus.cc:2781] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.424023 31549 raft_consensus.cc:686] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:34.424340 31549 consensus_queue.cc:237] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } }
I20240501 14:01:34.427356 29760 catalog_manager.cc:5484] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:34.489617 31498 raft_consensus.cc:482] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.489622 31549 raft_consensus.cc:482] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.489833 31498 raft_consensus.cc:504] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.489835 31549 raft_consensus.cc:504] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.490557 31498 leader_election.cc:288] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:34.491115 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:34.491057 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:34.491078 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:34.491163 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:34.491350 29900 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:34.491349 30032 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:34.491365 30292 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:34.491770 31549 leader_election.cc:288] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:34.491765 29967 leader_election.cc:302] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:34.492169 29837 leader_election.cc:302] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:34.492385 31549 raft_consensus.cc:2781] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.492478 31549 raft_consensus.cc:482] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.492466 31498 raft_consensus.cc:2781] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.492583 31498 raft_consensus.cc:482] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.492585 31549 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.492669 31498 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.494758 31549 raft_consensus.cc:504] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.494843 31498 raft_consensus.cc:504] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.495416 31549 leader_election.cc:288] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:34.495867 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:34.495862 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:34.495934 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:34.496106 30032 raft_consensus.cc:2379] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7869901e77234aeeb1462876b63eb28e in current term 1: Already voted for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in this term.
I20240501 14:01:34.496083 30292 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.496173 29900 raft_consensus.cc:2379] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in current term 1: Already voted for candidate 7869901e77234aeeb1462876b63eb28e in this term.
I20240501 14:01:34.496631 31498 leader_election.cc:288] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:34.497112 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:34.497731 29836 leader_election.cc:302] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e; no voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:34.498057 31498 raft_consensus.cc:2726] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:01:34.498566 30292 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:34.498970 29967 leader_election.cc:302] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 7869901e77234aeeb1462876b63eb28e
I20240501 14:01:34.499516 31549 raft_consensus.cc:2781] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.499806 31549 raft_consensus.cc:686] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:34.500398 31549 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } }
I20240501 14:01:34.503577 29760 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:34.556690 31492 raft_consensus.cc:482] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.556932 31492 raft_consensus.cc:504] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.557719 31492 leader_election.cc:288] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:34.558229 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3626b1c8d458e9c6d4a28bc0564fa" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:34.558377 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3626b1c8d458e9c6d4a28bc0564fa" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e" is_pre_election: true
I20240501 14:01:34.558463 30162 raft_consensus.cc:2454] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:34.558614 29900 raft_consensus.cc:2454] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:34.558907 30228 leader_election.cc:302] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:34.559221 31569 raft_consensus.cc:2781] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.559340 31569 raft_consensus.cc:482] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.559490 31569 raft_consensus.cc:3037] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.561650 31569 raft_consensus.cc:504] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.562219 31569 leader_election.cc:288] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:34.562659 29900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3626b1c8d458e9c6d4a28bc0564fa" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:34.562703 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3626b1c8d458e9c6d4a28bc0564fa" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:34.562829 29900 raft_consensus.cc:3037] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.562914 30162 raft_consensus.cc:3037] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.564774 29900 raft_consensus.cc:2454] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:34.564898 30162 raft_consensus.cc:2454] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:34.565361 30228 leader_election.cc:302] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:34.565694 31569 raft_consensus.cc:2781] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.565896 31569 raft_consensus.cc:686] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:34.566169 31569 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:34.569259 29760 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:34.890023 31549 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.896278 31549 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.899847 31037 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.916945 31037 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.965337 31569 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.971608 31492 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:35.043629 31572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:35.043975 31572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:35.051889 31572 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 235.051 ms
time per row: 0.117525 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:01:36.409175 29753 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54244:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:36.418712 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 64012de915564c7887e44d432b170fdc (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.418733 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 64012de915564c7887e44d432b170fdc (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.419185 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64012de915564c7887e44d432b170fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.419203 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64012de915564c7887e44d432b170fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.419334 30272 tablet_service.cc:1467] Processing CreateTablet for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.419765 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427af2a000f34ab6b9e3c9c7d7be1b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.424103 29880 tablet_service.cc:1467] Processing CreateTablet for tablet 212e139f1e7c4674987c2309efe3e495 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.424580 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212e139f1e7c4674987c2309efe3e495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.426363 30141 tablet_service.cc:1467] Processing CreateTablet for tablet 64012de915564c7887e44d432b170fdc (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.426462 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.426813 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64012de915564c7887e44d432b170fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.426777 30142 tablet_service.cc:1467] Processing CreateTablet for tablet 212e139f1e7c4674987c2309efe3e495 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.427058 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427af2a000f34ab6b9e3c9c7d7be1b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.427455 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.427772 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212e139f1e7c4674987c2309efe3e495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.427839 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427af2a000f34ab6b9e3c9c7d7be1b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.433579 30012 tablet_service.cc:1467] Processing CreateTablet for tablet 212e139f1e7c4674987c2309efe3e495 (DEFAULT_TABLE table=rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:36.434064 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212e139f1e7c4674987c2309efe3e495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:36.434228 31638 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:36.436578 31638 tablet_bootstrap.cc:654] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.438611 31640 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:36.442780 31639 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Bootstrap starting.
I20240501 14:01:36.443877 31640 tablet_bootstrap.cc:654] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.447528 31639 tablet_bootstrap.cc:654] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.447863 31638 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:36.448020 31638 ts_tablet_manager.cc:1367] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 14:01:36.448731 31638 raft_consensus.cc:348] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.448901 31638 raft_consensus.cc:374] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.449015 31638 raft_consensus.cc:729] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.449296 31638 consensus_queue.cc:260] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.449956 31638 ts_tablet_manager.cc:1398] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:36.450306 31638 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Bootstrap starting.
I20240501 14:01:36.451337 31639 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: No bootstrap required, opened a new log
I20240501 14:01:36.452175 31639 ts_tablet_manager.cc:1367] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:01:36.452203 31638 tablet_bootstrap.cc:654] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.452965 31639 raft_consensus.cc:348] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.454460 31639 raft_consensus.cc:374] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.454751 31639 raft_consensus.cc:729] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.455201 31639 consensus_queue.cc:260] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.456861 31639 ts_tablet_manager.cc:1398] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:36.458447 31641 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:36.459827 31638 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: No bootstrap required, opened a new log
I20240501 14:01:36.460005 31638 ts_tablet_manager.cc:1367] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 14:01:36.460763 31638 raft_consensus.cc:348] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.460999 31638 raft_consensus.cc:374] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.461093 31638 raft_consensus.cc:729] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.461409 31638 consensus_queue.cc:260] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.462014 31638 ts_tablet_manager.cc:1398] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:36.463153 31641 tablet_bootstrap.cc:654] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.463162 31640 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:36.463315 31640 ts_tablet_manager.cc:1367] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.005s
I20240501 14:01:36.464159 31640 raft_consensus.cc:348] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.464375 31640 raft_consensus.cc:374] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.464509 31640 raft_consensus.cc:729] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.464766 31640 consensus_queue.cc:260] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.465194 31641 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:36.465363 31640 ts_tablet_manager.cc:1398] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:01:36.465587 31641 ts_tablet_manager.cc:1367] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:01:36.465788 31640 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:36.466336 31641 raft_consensus.cc:348] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.466581 31641 raft_consensus.cc:374] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.466683 31641 raft_consensus.cc:729] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.466948 31641 consensus_queue.cc:260] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.467432 31640 tablet_bootstrap.cc:654] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.468523 31641 ts_tablet_manager.cc:1398] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:36.468883 31641 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:36.471096 31641 tablet_bootstrap.cc:654] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.473835 31640 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:36.473910 31641 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:36.474041 31640 ts_tablet_manager.cc:1367] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240501 14:01:36.474121 31641 ts_tablet_manager.cc:1367] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:01:36.474886 31640 raft_consensus.cc:348] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.474898 31641 raft_consensus.cc:348] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.475129 31641 raft_consensus.cc:374] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.475193 31640 raft_consensus.cc:374] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.475281 31641 raft_consensus.cc:729] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.475358 31640 raft_consensus.cc:729] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.475567 31641 consensus_queue.cc:260] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.475617 31640 consensus_queue.cc:260] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.476117 31641 ts_tablet_manager.cc:1398] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:36.476162 31640 ts_tablet_manager.cc:1398] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:36.476409 31641 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Bootstrap starting.
I20240501 14:01:36.477340 31640 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812: Bootstrap starting.
I20240501 14:01:36.478626 31640 tablet_bootstrap.cc:654] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.479987 31641 tablet_bootstrap.cc:654] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:36.482153 31641 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: No bootstrap required, opened a new log
I20240501 14:01:36.482333 31641 ts_tablet_manager.cc:1367] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:01:36.483207 31641 raft_consensus.cc:348] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.483381 31641 raft_consensus.cc:374] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.483472 31641 raft_consensus.cc:729] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.483666 31641 consensus_queue.cc:260] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.483745 31640 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812: No bootstrap required, opened a new log
I20240501 14:01:36.483879 31640 ts_tablet_manager.cc:1367] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:01:36.484005 31608 raft_consensus.cc:482] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.484205 31641 ts_tablet_manager.cc:1398] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:36.484175 31608 raft_consensus.cc:504] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.484669 31640 raft_consensus.cc:348] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.484869 31640 raft_consensus.cc:374] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:36.484952 31608 leader_election.cc:288] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.484966 31640 raft_consensus.cc:729] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Initialized, Role: FOLLOWER
I20240501 14:01:36.485215 31640 consensus_queue.cc:260] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.485780 31640 ts_tablet_manager.cc:1398] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:36.486106 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212e139f1e7c4674987c2309efe3e495" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:36.486322 30032 raft_consensus.cc:2454] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:36.486795 29837 leader_election.cc:302] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:36.487309 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212e139f1e7c4674987c2309efe3e495" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:36.487527 30160 raft_consensus.cc:2454] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 0.
I20240501 14:01:36.487658 31608 raft_consensus.cc:2781] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.487776 31608 raft_consensus.cc:482] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.487865 31608 raft_consensus.cc:3037] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.490316 31608 raft_consensus.cc:504] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.490849 31608 leader_election.cc:288] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.491340 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212e139f1e7c4674987c2309efe3e495" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:36.491560 30032 raft_consensus.cc:3037] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.491546 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212e139f1e7c4674987c2309efe3e495" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:36.491719 30160 raft_consensus.cc:3037] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.493718 30032 raft_consensus.cc:2454] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:36.493862 30160 raft_consensus.cc:2454] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 1.
I20240501 14:01:36.494338 29837 leader_election.cc:302] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:36.494637 31608 raft_consensus.cc:2781] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.494779 31608 raft_consensus.cc:686] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:36.495079 31608 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.498436 29759 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:36.662824 31418 raft_consensus.cc:482] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.663033 31418 raft_consensus.cc:504] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.663730 31570 raft_consensus.cc:482] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.663921 31418 leader_election.cc:288] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:36.663905 31570 raft_consensus.cc:504] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.664638 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:36.664608 31570 leader_election.cc:288] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.664885 30292 raft_consensus.cc:2454] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:36.665060 30030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:36.665040 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:36.665321 30030 raft_consensus.cc:2454] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:36.665343 30160 raft_consensus.cc:2454] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 0.
I20240501 14:01:36.665397 30097 leader_election.cc:302] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:36.664602 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" is_pre_election: true
I20240501 14:01:36.665702 30032 raft_consensus.cc:2454] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 0.
I20240501 14:01:36.665742 31418 raft_consensus.cc:2781] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.665771 30228 leader_election.cc:302] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812; no voters:
I20240501 14:01:36.665853 31418 raft_consensus.cc:482] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.665941 31418 raft_consensus.cc:3037] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.666258 31570 raft_consensus.cc:2781] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.666381 31570 raft_consensus.cc:482] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.666471 31570 raft_consensus.cc:3037] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.668885 31418 raft_consensus.cc:504] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.669108 31570 raft_consensus.cc:504] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.669698 31418 leader_election.cc:288] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:36.669795 31570 leader_election.cc:288] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.670339 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:36.670327 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:36.670410 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:36.670367 30030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64012de915564c7887e44d432b170fdc" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:36.670588 30292 raft_consensus.cc:2379] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76c7a691de8a4e31b32799f43cb4b812 in current term 1: Already voted for candidate 4a15e881772b41eab8a71fc027390db9 in this term.
I20240501 14:01:36.670630 30030 raft_consensus.cc:3037] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.670746 30160 raft_consensus.cc:2379] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a15e881772b41eab8a71fc027390db9 in current term 1: Already voted for candidate 76c7a691de8a4e31b32799f43cb4b812 in this term.
I20240501 14:01:36.671630 30098 leader_election.cc:302] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812; no voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:01:36.671943 31418 raft_consensus.cc:2726] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:01:36.673071 30030 raft_consensus.cc:2454] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 1.
I20240501 14:01:36.673493 30228 leader_election.cc:302] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters: 76c7a691de8a4e31b32799f43cb4b812
I20240501 14:01:36.676558 31570 raft_consensus.cc:2781] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.676998 31570 raft_consensus.cc:686] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:36.677398 31570 consensus_queue.cc:237] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.681352 29759 catalog_manager.cc:5484] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:36.793675 31632 raft_consensus.cc:482] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.793885 31632 raft_consensus.cc:504] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.794575 31632 leader_election.cc:288] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.795082 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427af2a000f34ab6b9e3c9c7d7be1b32" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812" is_pre_election: true
I20240501 14:01:36.795141 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427af2a000f34ab6b9e3c9c7d7be1b32" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9" is_pre_election: true
I20240501 14:01:36.795357 30160 raft_consensus.cc:2454] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:36.795400 30292 raft_consensus.cc:2454] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 0.
I20240501 14:01:36.795855 29968 leader_election.cc:302] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:36.796247 31632 raft_consensus.cc:2781] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.796377 31632 raft_consensus.cc:482] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.796469 31632 raft_consensus.cc:3037] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.798573 31632 raft_consensus.cc:504] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.799293 31632 leader_election.cc:288] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:36.799719 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427af2a000f34ab6b9e3c9c7d7be1b32" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:36.799793 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427af2a000f34ab6b9e3c9c7d7be1b32" candidate_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
I20240501 14:01:36.799913 30292 raft_consensus.cc:3037] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.800025 30160 raft_consensus.cc:3037] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.802927 30160 raft_consensus.cc:2454] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:36.802939 30292 raft_consensus.cc:2454] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac7d9aa0b5e49de9962e9067d2bc0e4 in term 1.
I20240501 14:01:36.803526 29967 leader_election.cc:302] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:36.804047 31654 raft_consensus.cc:2781] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.804216 31654 raft_consensus.cc:686] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:36.804513 31654 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } }
I20240501 14:01:36.809746 29759 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:37.041481 31633 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.047181 31633 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.059780 31570 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.067685 31570 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:37.286033 31657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:37.286370 31657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:37.294051 31657 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:01:37.402942 31654 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.424624 31689 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:37.516534 29968 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:38823, user_credentials={real_user=slave}} blocked reactor thread for 93820.7us
INSERT report
rows total: 2000
time total: 343.817 ms
time per row: 0.171908 ms
I20240501 14:01:37.797360 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.197:34471
--local_ip_for_outbound_sockets=127.22.5.197
--tserver_master_addrs=127.22.5.254:36941
--webserver_port=35409
--webserver_interface=127.22.5.197
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:01:38.187134 31731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:38.187549 31731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:38.187718 31731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:38.198177 31731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:38.198328 31731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:38.198590 31731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.197
I20240501 14:01:38.209342 31731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.197:34471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.5.197
--webserver_port=35409
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.197
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:38.210733 31731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:38.212980 31731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:38.226552 31736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:38.227066 31737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:38.227582 31739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:38.228216 31731 server_base.cc:1034] running on GCE node
I20240501 14:01:38.228900 31731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:38.229959 31731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:38.231211 31731 hybrid_clock.cc:648] HybridClock initialized: now 1714572098231156 us; error 48 us; skew 500 ppm
I20240501 14:01:38.231537 31731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:38.233914 31731 webserver.cc:460] Webserver started at http://127.22.5.197:35409/ using document root <none> and password file <none>
I20240501 14:01:38.234671 31731 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:38.234807 31731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:38.241093 31731 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:01:38.244057 31745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.245558 31731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:38.245703 31731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:38.246743 31731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:38.264775 31731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:38.265790 31731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:38.266137 31731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:38.267522 31731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:38.269145 31731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:38.269246 31731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.269373 31731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:38.269430 31731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.307255 31731 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.197:34471
I20240501 14:01:38.307264 31857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.197:34471 every 8 connection(s)
I20240501 14:01:38.309459 31731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:01:38.317945 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 31731
I20240501 14:01:38.318598 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.198:36301
--local_ip_for_outbound_sockets=127.22.5.198
--tserver_master_addrs=127.22.5.254:36941
--webserver_port=46557
--webserver_interface=127.22.5.198
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:38.324250 31858 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:38.324571 31858 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:38.325562 31858 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:38.327376 29758 ts_manager.cc:194] Re-registered known tserver with Master: be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
I20240501 14:01:38.328609 29758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.197:49543
W20240501 14:01:38.696475 31862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:38.696820 31862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:38.696981 31862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:38.705420 31862 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:38.705569 31862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:38.705724 31862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.198
I20240501 14:01:38.715564 31862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.198:36301
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.5.198
--webserver_port=46557
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.198
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:38.717226 31862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:38.719437 31862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:38.732846 31872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:38.733353 31870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:38.733742 31869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:38.734565 31862 server_base.cc:1034] running on GCE node
I20240501 14:01:38.735283 31862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:38.736407 31862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:38.737625 31862 hybrid_clock.cc:648] HybridClock initialized: now 1714572098737569 us; error 54 us; skew 500 ppm
I20240501 14:01:38.738095 31862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:38.740942 31862 webserver.cc:460] Webserver started at http://127.22.5.198:46557/ using document root <none> and password file <none>
I20240501 14:01:38.741796 31862 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:38.741919 31862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:38.747815 31862 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:01:38.751055 31878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.752861 31862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20240501 14:01:38.753050 31862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3321c47e58ba49f09f4f036ac72fc133"
format_stamp: "Formatted at 2024-05-01 14:01:19 on dist-test-slave-5jzj"
I20240501 14:01:38.754060 31862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:38.771528 31862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:38.772605 31862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:38.772960 31862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:38.774005 31862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:38.775570 31862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:38.775643 31862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.775763 31862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:38.775827 31862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:38.810922 31862 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.198:36301
I20240501 14:01:38.810957 31990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.198:36301 every 8 connection(s)
I20240501 14:01:38.812377 31862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:01:38.818137 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 31862
I20240501 14:01:38.818760 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.199:40447
--local_ip_for_outbound_sockets=127.22.5.199
--tserver_master_addrs=127.22.5.254:36941
--webserver_port=39397
--webserver_interface=127.22.5.199
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:38.825945 31991 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:38.826279 31991 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:38.827150 31991 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:38.828652 29758 ts_manager.cc:194] Re-registered known tserver with Master: 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
I20240501 14:01:38.829631 29758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.198:56147
W20240501 14:01:39.202828 31994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:39.203279 31994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:39.203426 31994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:39.214695 31994 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:39.214823 31994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:39.214953 31994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.199
I20240501 14:01:39.224247 31994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.199:40447
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.5.199
--webserver_port=39397
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.199
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:39.225605 31994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:39.227293 31994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:39.240410 32001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:39.240569 32003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:39.241209 32000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:39.241823 31994 server_base.cc:1034] running on GCE node
I20240501 14:01:39.242484 31994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:39.243716 31994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:39.244943 31994 hybrid_clock.cc:648] HybridClock initialized: now 1714572099244848 us; error 97 us; skew 500 ppm
I20240501 14:01:39.245319 31994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:39.247694 31994 webserver.cc:460] Webserver started at http://127.22.5.199:39397/ using document root <none> and password file <none>
I20240501 14:01:39.248457 31994 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:39.248570 31994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:39.255203 31994 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:01:39.258720 32009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.260304 31994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:01:39.260471 31994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:39.261382 31994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:39.297380 31994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:39.298374 31994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:39.298709 31994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:39.299733 31994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:39.301492 31994 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:39.301584 31994 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.301716 31994 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:39.301776 31994 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.331444 31858 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:39.340814 32121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.199:40447 every 8 connection(s)
I20240501 14:01:39.340809 31994 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.199:40447
I20240501 14:01:39.342499 31994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:01:39.350035 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 31994
I20240501 14:01:39.350791 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.200:43981
--local_ip_for_outbound_sockets=127.22.5.200
--tserver_master_addrs=127.22.5.254:36941
--webserver_port=42041
--webserver_interface=127.22.5.200
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:39.357259 32122 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:39.357594 32122 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:39.358561 32122 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:39.360162 29758 ts_manager.cc:194] Re-registered known tserver with Master: 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447)
I20240501 14:01:39.361341 29758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.199:57685
W20240501 14:01:39.730983 32126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:39.731508 32126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:39.731721 32126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:39.739729 32126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:39.739874 32126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:39.740053 32126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.200
I20240501 14:01:39.750816 32126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.200:43981
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.5.200
--webserver_port=42041
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.200
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:39.752324 32126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:39.754148 32126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:39.767809 32133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:39.768311 32134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:39.768735 32136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:39.769409 32126 server_base.cc:1034] running on GCE node
I20240501 14:01:39.770015 32126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:39.771040 32126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:39.772245 32126 hybrid_clock.cc:648] HybridClock initialized: now 1714572099772187 us; error 61 us; skew 500 ppm
I20240501 14:01:39.772593 32126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:39.775008 32126 webserver.cc:460] Webserver started at http://127.22.5.200:42041/ using document root <none> and password file <none>
I20240501 14:01:39.775763 32126 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:39.775880 32126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:39.781677 32126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:01:39.784837 32142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.786370 32126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:39.786514 32126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "744cef852ee9491db3928fa0092ef189"
format_stamp: "Formatted at 2024-05-01 14:01:20 on dist-test-slave-5jzj"
I20240501 14:01:39.787314 32126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:39.825882 32126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:39.826767 32126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:39.827029 32126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:39.828006 32126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:39.829636 32126 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:39.829715 32126 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.829818 32126 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:39.829861 32126 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:39.832211 31991 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:39.864737 32126 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.200:43981
I20240501 14:01:39.864775 32254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.200:43981 every 8 connection(s)
I20240501 14:01:39.866353 32126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:01:39.873966 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 32126
I20240501 14:01:39.881472 32255 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:39.881927 32255 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:39.882923 32255 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:39.884491 29758 ts_manager.cc:194] Re-registered known tserver with Master: 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:01:39.885756 29758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.200:51111
W20240501 14:01:40.270149 32261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:40.270655 32261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:40.278684 32261 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:40.364289 32122 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:40.509672 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.518162 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:40.552381 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.561591 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.570170 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.592803 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:40.604712 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.628585 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:40.652917 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:40.697175 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:40.710039 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.722462 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.728783 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:40.777614 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:40.786324 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:40.825274 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:40.888583 32255 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:41.109705 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.112740 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.117467 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:41.137528 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.156946 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:41.163095 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:41.199003 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:41.235718 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.513885 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.549722 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.559317 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:41.559573 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:41.571264 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:41.579770 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:41.591885 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:41.609869 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:41.695765 32384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:41.696175 32384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:41.712317 32384 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:41.786585 30032 consensus_queue.cc:237] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:41.790874 30162 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.790923 29899 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.791607 32127 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:41.791985 32451 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:41.795641 32127 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:41.797952 29899 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:41.805553 29754 catalog_manager.cc:5484] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:41.808763 29965 consensus_peers.cc:485] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a733857ad51d452a9054cc5f533edc8e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:41.834064 30162 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:41.846024 30292 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:41.851449 30162 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.852123 30032 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.852681 32489 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:41.853957 31570 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:41.860705 30227 consensus_peers.cc:485] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c40a79b118424a5699bddd1c4ac4bb0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:41.863520 32489 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:41.864698 30030 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:41.865248 30160 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:41.871615 29756 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:41.913818 29899 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:41.921258 30162 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.937551 30030 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:41.937551 32509 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:41.939677 32509 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:41.952138 29837 consensus_peers.cc:485] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 368edc181c894686a213fc542f328192. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:41.958537 31995 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:41.959614 30030 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:41.966277 29752 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:41.972975 30162 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:41.995514 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:42.009058 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:42.011906 29899 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:42.021977 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:42.023654 30292 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.024430 30162 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.024827 31995 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.026546 31995 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.065168 30161 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.065168 30290 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.062472 31995 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.073130 29752 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.075982 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:42.093173 29837 consensus_peers.cc:485] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c35ccca22fb45b78ee66991027c0261. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.112821 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:42.114933 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:42.118327 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:42.125275 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:42.170758 30161 consensus_queue.cc:237] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:42.213990 30290 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.214670 29901 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.214676 32416 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.215308 32416 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.223901 32540 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:42.224829 30290 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
W20240501 14:01:42.231462 30095 consensus_peers.cc:485] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc0201075e1466a9661b155c180d95f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.236532 29901 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:42.237152 29756 catalog_manager.cc:5484] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
Runtime error: ksck discovered errors
I20240501 14:01:42.265209 30030 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:42.271942 30290 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.273586 32550 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.274386 30161 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.275028 32451 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.278662 32451 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
W20240501 14:01:42.283762 29967 consensus_peers.cc:485] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 427af2a000f34ab6b9e3c9c7d7be1b32. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.284139 30290 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.288060 29752 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:42.296175 30161 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.346112 32558 ts_tablet_manager.cc:902] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:42.348058 30161 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:42.355398 30030 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.356113 32416 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.361709 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.369107 32416 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:42.373735 30098 consensus_peers.cc:485] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 642312cdad114fce839d2b60717bcc72. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.386018 32563 ts_tablet_manager.cc:902] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:42.394714 32540 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.412631 32564 ts_tablet_manager.cc:902] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:42.413630 32563 tablet_copy_client.cc:308] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:42.416167 30290 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.416749 30030 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.428233 29752 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.435288 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 368edc181c894686a213fc542f328192 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:37341)
I20240501 14:01:42.435585 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 368edc181c894686a213fc542f328192 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:37341: session id = 3321c47e58ba49f09f4f036ac72fc133-368edc181c894686a213fc542f328192
I20240501 14:01:42.439612 29921 tablet_copy_source_session.cc:205] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.445991 32563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368edc181c894686a213fc542f328192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.462987 32563 tablet_copy_client.cc:783] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.463338 32563 tablet_copy_client.cc:647] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.466220 32570 tablet_copy_client.cc:1002] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.478546 32558 tablet_copy_client.cc:308] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:42.482992 29901 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:42.486403 30290 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:42.487429 29837 consensus_peers.cc:485] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ea9aaebd437424eb27c04bc2f6ffa7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.487532 32509 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.489257 30030 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.489970 32509 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.507586 31995 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.514020 30292 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.522251 29752 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.535151 32584 ts_tablet_manager.cc:902] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:42.536878 32564 tablet_copy_client.cc:308] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:42.544234 30030 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.563186 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet c40a79b118424a5699bddd1c4ac4bb0e from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:33017)
I20240501 14:01:42.563515 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet c40a79b118424a5699bddd1c4ac4bb0e from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:33017: session id = 47b7351d2e9b4dbd9a09fd4133a97565-c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:01:42.568033 30312 tablet_copy_source_session.cc:205] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.574594 32558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a79b118424a5699bddd1c4ac4bb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.579669 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet a733857ad51d452a9054cc5f533edc8e from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:39969)
I20240501 14:01:42.579977 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet a733857ad51d452a9054cc5f533edc8e from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:39969: session id = be55dad80e3a4ad79d21aaa3199b1d0b-a733857ad51d452a9054cc5f533edc8e
I20240501 14:01:42.585537 30052 tablet_copy_source_session.cc:205] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.589268 32558 tablet_copy_client.cc:783] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.589692 32558 tablet_copy_client.cc:647] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.591727 32564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a733857ad51d452a9054cc5f533edc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.592721 32582 tablet_copy_client.cc:1002] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.613036 32564 tablet_copy_client.cc:783] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.613513 32564 tablet_copy_client.cc:647] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.621143 32587 tablet_copy_client.cc:1002] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.626225 30290 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:42.629420 29901 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:42.629830 30227 consensus_peers.cc:485] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7d3626b1c8d458e9c6d4a28bc0564fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.630174 32489 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.630492 30160 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.631062 32489 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.633461 32584 tablet_copy_client.cc:308] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:42.662932 32492 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.665956 29899 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.677788 29752 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.705440 30160 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:42.711907 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 5c35ccca22fb45b78ee66991027c0261 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:42889)
I20240501 14:01:42.712121 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 5c35ccca22fb45b78ee66991027c0261 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:42889: session id = 744cef852ee9491db3928fa0092ef189-5c35ccca22fb45b78ee66991027c0261
I20240501 14:01:42.717048 29921 tablet_copy_source_session.cc:205] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.721999 32584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c35ccca22fb45b78ee66991027c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.737401 32584 tablet_copy_client.cc:783] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.737883 32584 tablet_copy_client.cc:647] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.739935 32602 tablet_copy_client.cc:1002] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.757691 32612 ts_tablet_manager.cc:902] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:42.762789 32612 tablet_copy_client.cc:308] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:42.770821 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:46389)
I20240501 14:01:42.771059 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 427af2a000f34ab6b9e3c9c7d7be1b32 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:46389: session id = 47b7351d2e9b4dbd9a09fd4133a97565-427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:01:42.772168 30290 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:42.773324 30052 tablet_copy_source_session.cc:205] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.775400 32612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427af2a000f34ab6b9e3c9c7d7be1b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.775580 30160 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.776309 32489 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.783206 29899 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.782891 32489 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:42.784657 32612 tablet_copy_client.cc:783] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.784914 32612 tablet_copy_client.cc:647] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.785216 32492 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.787503 32617 tablet_copy_client.cc:1002] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.787587 30160 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:42.791569 29752 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:42.814342 30225 consensus_peers.cc:485] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56f632889b1e400392fe710bf9f6f3ec. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.817641 29899 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:42.821914 32624 ts_tablet_manager.cc:902] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:42.844345 32624 tablet_copy_client.cc:308] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:42.852083 32634 ts_tablet_manager.cc:902] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:42.853420 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 4cc0201075e1466a9661b155c180d95f from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:45235)
I20240501 14:01:42.853749 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 4cc0201075e1466a9661b155c180d95f from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:45235: session id = be55dad80e3a4ad79d21aaa3199b1d0b-4cc0201075e1466a9661b155c180d95f
I20240501 14:01:42.854990 29899 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:42.857641 30182 tablet_copy_source_session.cc:205] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.857717 30290 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.858424 30030 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.859083 31995 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.860484 32591 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.862424 32624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc0201075e1466a9661b155c180d95f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:01:42.863272 29837 consensus_peers.cc:485] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 582bddb40ece4d5b8dcd3e1c00a53536. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.870584 32624 tablet_copy_client.cc:783] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.870865 32624 tablet_copy_client.cc:647] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.873189 32631 tablet_copy_client.cc:1002] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.875994 32591 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.880319 30032 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.889684 29752 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.896881 30292 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:42.902433 32634 tablet_copy_client.cc:308] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:42.904007 32640 ts_tablet_manager.cc:902] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:42.916894 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 642312cdad114fce839d2b60717bcc72 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:39861)
I20240501 14:01:42.917171 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 642312cdad114fce839d2b60717bcc72 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:39861: session id = 744cef852ee9491db3928fa0092ef189-642312cdad114fce839d2b60717bcc72
I20240501 14:01:42.922175 30182 tablet_copy_source_session.cc:205] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.926096 32634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642312cdad114fce839d2b60717bcc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.932828 32634 tablet_copy_client.cc:783] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.933140 32634 tablet_copy_client.cc:647] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.934731 32643 tablet_copy_client.cc:1002] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.938078 32640 tablet_copy_client.cc:308] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:42.939199 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:37341)
I20240501 14:01:42.939361 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 0ea9aaebd437424eb27c04bc2f6ffa7b from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:37341: session id = 3321c47e58ba49f09f4f036ac72fc133-0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:01:42.942312 29921 tablet_copy_source_session.cc:205] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:42.943578 32640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.946264 29899 consensus_queue.cc:237] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:42.950803 32640 tablet_copy_client.cc:783] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:42.951046 32640 tablet_copy_client.cc:647] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:42.954934 30292 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:42.955579 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.958938 32513 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.959514 30292 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.963256 30160 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:42.964679 29837 consensus_peers.cc:485] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c953ab40905a4d23b9b9c970d25168b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:42.964890 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:42.964972 32651 tablet_copy_client.cc:1002] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:42.966657 29757 catalog_manager.cc:5484] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:42.978408 30160 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:42.996291 30032 consensus_queue.cc:237] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:42.999667 29899 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.001420 30160 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.001813 32550 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.002249 32451 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.006520 32451 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.007272 30160 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.015714 29757 catalog_manager.cc:5484] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.019402 29899 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
W20240501 14:01:43.021968 29968 consensus_peers.cc:485] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9735c5d9f784459ebafd8f2ff4f69763. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.041085 32668 ts_tablet_manager.cc:902] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:43.051255 32668 tablet_copy_client.cc:308] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:43.052129 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet f7d3626b1c8d458e9c6d4a28bc0564fa from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:33017)
I20240501 14:01:43.052315 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet f7d3626b1c8d458e9c6d4a28bc0564fa from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:33017: session id = 47b7351d2e9b4dbd9a09fd4133a97565-f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:01:43.055126 30312 tablet_copy_source_session.cc:205] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.056543 32668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3626b1c8d458e9c6d4a28bc0564fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.060233 29899 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:43.063024 32668 tablet_copy_client.cc:783] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.063292 32668 tablet_copy_client.cc:647] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.063787 30160 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.064407 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.065017 32675 tablet_copy_client.cc:1002] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.065166 32509 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.066898 32509 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:43.077095 29836 consensus_peers.cc:485] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef202920b8d64287a8a2cdb40c73a819. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.077466 32509 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.082166 30031 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.086409 30160 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.087492 29752 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.113075 29899 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
W20240501 14:01:43.113487 32583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:43.113909 32583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:43.116235 30032 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.116281 30160 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.117645 32513 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.118049 32509 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.122098 32509 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
W20240501 14:01:43.124490 32583 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240501 14:01:43.125989 29834 consensus_peers.cc:485] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 212e139f1e7c4674987c2309efe3e495. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.126567 30160 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.126737 30032 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.131237 29756 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.154552 30292 consensus_queue.cc:237] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:43.157774 30160 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.158402 30032 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.159371 32604 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.159731 32489 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:43.164767 30228 consensus_peers.cc:485] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edf4bcf02c7e4352bd998e139802db62. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.177695 32604 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.181216 30032 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.183071 29756 catalog_manager.cc:5484] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.183424 30160 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:43.260622 30160 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:43.264560 29899 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.265596 30290 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.265533 32723 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.266216 32568 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.272409 32568 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:43.273075 30290 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:43.279279 29752 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.285224 29899 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
W20240501 14:01:43.290321 30098 consensus_peers.cc:485] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98d94ed11f0f4a9d9f519a24c3320ce4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.315563 32740 ts_tablet_manager.cc:902] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:43.316731 30032 consensus_queue.cc:237] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
W20240501 14:01:43.342104 29965 consensus_peers.cc:485] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1bde99cadd94efc826b4074b2674104. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.343196 29899 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.343830 32665 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.351292 32741 ts_tablet_manager.cc:902] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:43.354964 30160 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.360998 32665 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.383342 32665 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.385969 30161 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.393572 29752 catalog_manager.cc:5484] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.411991 32750 ts_tablet_manager.cc:902] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:43.416061 29901 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:43.439121 32740 tablet_copy_client.cc:308] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:43.445590 32741 tablet_copy_client.cc:308] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:43.446334 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 582bddb40ece4d5b8dcd3e1c00a53536 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:42889)
I20240501 14:01:43.446533 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 582bddb40ece4d5b8dcd3e1c00a53536 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:42889: session id = 744cef852ee9491db3928fa0092ef189-582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:01:43.450647 29921 tablet_copy_source_session.cc:205] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.453593 32741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bddb40ece4d5b8dcd3e1c00a53536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.463989 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet 56f632889b1e400392fe710bf9f6f3ec from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:37661)
I20240501 14:01:43.464160 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet 56f632889b1e400392fe710bf9f6f3ec from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:37661: session id = be55dad80e3a4ad79d21aaa3199b1d0b-56f632889b1e400392fe710bf9f6f3ec
I20240501 14:01:43.467087 30312 tablet_copy_source_session.cc:205] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.467285 32750 tablet_copy_client.cc:308] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:43.468573 32740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f632889b1e400392fe710bf9f6f3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.468822 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet c953ab40905a4d23b9b9c970d25168b9 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:37341)
I20240501 14:01:43.468991 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet c953ab40905a4d23b9b9c970d25168b9 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:37341: session id = 3321c47e58ba49f09f4f036ac72fc133-c953ab40905a4d23b9b9c970d25168b9
I20240501 14:01:43.469622 32741 tablet_copy_client.cc:783] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.469851 32741 tablet_copy_client.cc:647] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.471514 32760 tablet_copy_client.cc:1002] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.472992 29921 tablet_copy_source_session.cc:205] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.477615 32750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c953ab40905a4d23b9b9c970d25168b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.477843 32740 tablet_copy_client.cc:783] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.478194 32740 tablet_copy_client.cc:647] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.481681 32759 tablet_copy_client.cc:1002] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.486532 32750 tablet_copy_client.cc:783] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.486820 32750 tablet_copy_client.cc:647] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.487269 32767 ts_tablet_manager.cc:902] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:43.489014 32764 tablet_copy_client.cc:1002] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.490448 300 ts_tablet_manager.cc:902] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:43.504544 300 tablet_copy_client.cc:308] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:43.509665 30161 consensus_queue.cc:237] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:43.520000 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:43.520675 30032 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.521389 32730 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.521914 30290 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:43.522472 32730 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:43.537763 32730 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
W20240501 14:01:43.545019 30097 consensus_peers.cc:485] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6241869f8e7f4bfaa65508fd61f405a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:43.547479 32057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:43.549963 32767 tablet_copy_client.cc:308] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:43.561558 29752 catalog_manager.cc:5484] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.563500 30290 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.576967 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet ef202920b8d64287a8a2cdb40c73a819 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:57317)
I20240501 14:01:43.577256 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet ef202920b8d64287a8a2cdb40c73a819 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:57317: session id = 47b7351d2e9b4dbd9a09fd4133a97565-ef202920b8d64287a8a2cdb40c73a819
I20240501 14:01:43.580848 29921 tablet_copy_source_session.cc:205] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.586257 32767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef202920b8d64287a8a2cdb40c73a819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.590955 30270 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:43.593731 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 9735c5d9f784459ebafd8f2ff4f69763 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:36943)
I20240501 14:01:43.593938 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 9735c5d9f784459ebafd8f2ff4f69763 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:36943: session id = 744cef852ee9491db3928fa0092ef189-9735c5d9f784459ebafd8f2ff4f69763
I20240501 14:01:43.597074 30052 tablet_copy_source_session.cc:205] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.598467 32767 tablet_copy_client.cc:783] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.598812 300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9735c5d9f784459ebafd8f2ff4f69763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.609030 300 tablet_copy_client.cc:783] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.609259 32767 tablet_copy_client.cc:647] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.609872 300 tablet_copy_client.cc:647] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.611430 315 tablet_copy_client.cc:1002] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.612675 308 tablet_copy_client.cc:1002] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.620581 30031 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:43.629616 321 ts_tablet_manager.cc:902] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:43.709650 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:43.713758 30141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:43.714519 324 ts_tablet_manager.cc:902] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:43.746217 30010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:01:43.746695 338 ts_tablet_manager.cc:902] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:43.773224 324 tablet_copy_client.cc:308] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:43.783277 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 212e139f1e7c4674987c2309efe3e495 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:38005)
I20240501 14:01:43.783463 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 212e139f1e7c4674987c2309efe3e495 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:38005: session id = be55dad80e3a4ad79d21aaa3199b1d0b-212e139f1e7c4674987c2309efe3e495
I20240501 14:01:43.786515 29921 tablet_copy_source_session.cc:205] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.788048 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212e139f1e7c4674987c2309efe3e495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.791461 321 tablet_copy_client.cc:308] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:43.792431 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:43.795447 29880 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:43.803735 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet edf4bcf02c7e4352bd998e139802db62 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:49589)
I20240501 14:01:43.803969 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet edf4bcf02c7e4352bd998e139802db62 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:49589: session id = 744cef852ee9491db3928fa0092ef189-edf4bcf02c7e4352bd998e139802db62
I20240501 14:01:43.805691 338 tablet_copy_client.cc:308] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:43.806456 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet d1bde99cadd94efc826b4074b2674104 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:39969)
I20240501 14:01:43.806638 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet d1bde99cadd94efc826b4074b2674104 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:39969: session id = be55dad80e3a4ad79d21aaa3199b1d0b-d1bde99cadd94efc826b4074b2674104
I20240501 14:01:43.807929 30312 tablet_copy_source_session.cc:205] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.810423 30052 tablet_copy_source_session.cc:205] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.815229 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1bde99cadd94efc826b4074b2674104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.825457 321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf4bcf02c7e4352bd998e139802db62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.836280 324 tablet_copy_client.cc:783] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.836699 324 tablet_copy_client.cc:647] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.843613 338 tablet_copy_client.cc:783] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.844029 338 tablet_copy_client.cc:647] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.846151 348 tablet_copy_client.cc:1002] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.851172 321 tablet_copy_client.cc:783] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.851549 321 tablet_copy_client.cc:647] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.853269 339 tablet_copy_client.cc:1002] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.878937 345 tablet_copy_client.cc:1002] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.892259 344 ts_tablet_manager.cc:902] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:43.909798 344 tablet_copy_client.cc:308] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:43.924528 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:60129)
I20240501 14:01:43.924738 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 98d94ed11f0f4a9d9f519a24c3320ce4 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:60129: session id = 3321c47e58ba49f09f4f036ac72fc133-98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:01:43.928093 30182 tablet_copy_source_session.cc:205] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.929819 344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d94ed11f0f4a9d9f519a24c3320ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.938957 344 tablet_copy_client.cc:783] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.939234 344 tablet_copy_client.cc:647] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:43.940943 362 tablet_copy_client.cc:1002] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:43.949752 370 ts_tablet_manager.cc:902] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:43.959542 370 tablet_copy_client.cc:308] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:43.966856 32563 tablet_copy_client.cc:523] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:43.971920 32563 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
Runtime error: ksck discovered errors
I20240501 14:01:43.973831 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 6241869f8e7f4bfaa65508fd61f405a6 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:45965)
I20240501 14:01:43.973982 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 6241869f8e7f4bfaa65508fd61f405a6 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:45965: session id = 47b7351d2e9b4dbd9a09fd4133a97565-6241869f8e7f4bfaa65508fd61f405a6
I20240501 14:01:43.975986 30182 tablet_copy_source_session.cc:205] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:43.982159 370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6241869f8e7f4bfaa65508fd61f405a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.992383 32563 log.cc:824] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:43.999581 370 tablet_copy_client.cc:783] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:43.999887 370 tablet_copy_client.cc:647] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:44.001907 378 tablet_copy_client.cc:1002] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:44.058671 32563 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.059391 32563 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:44.059903 32563 ts_tablet_manager.cc:1367] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.088s user 0.061s sys 0.013s
I20240501 14:01:44.066383 32563 raft_consensus.cc:348] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:44.066936 32563 raft_consensus.cc:729] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:44.067548 32563 consensus_queue.cc:260] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:44.069417 32563 ts_tablet_manager.cc:1398] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240501 14:01:44.070834 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-368edc181c894686a213fc542f328192 received from {username='slave'} at 127.22.5.198:37341
I20240501 14:01:44.071000 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-368edc181c894686a213fc542f328192 on tablet 368edc181c894686a213fc542f328192 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:44.093349 32558 tablet_copy_client.cc:523] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.098696 32558 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:44.113324 32558 log.cc:824] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:44.114506 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.201:0
--local_ip_for_outbound_sockets=127.22.5.201
--webserver_interface=127.22.5.201
--webserver_port=0
--tserver_master_addrs=127.22.5.254:36941
--builtin_ntp_servers=127.22.5.212:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:01:44.121711 32564 tablet_copy_client.cc:523] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.126507 32564 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:44.140290 32564 log.cc:824] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:44.207829 32558 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.210981 32558 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:44.218849 32564 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.219537 32564 tablet_bootstrap.cc:492] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:44.219539 32558 ts_tablet_manager.cc:1367] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.121s user 0.058s sys 0.013s
I20240501 14:01:44.219981 32564 ts_tablet_manager.cc:1367] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.094s user 0.074s sys 0.003s
I20240501 14:01:44.227465 32558 raft_consensus.cc:348] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.227432 32564 raft_consensus.cc:348] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:44.228089 32564 raft_consensus.cc:729] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:44.228173 32558 raft_consensus.cc:729] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:44.228672 32564 consensus_queue.cc:260] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:44.228899 32558 consensus_queue.cc:260] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.240478 32584 tablet_copy_client.cc:523] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.243078 32564 ts_tablet_manager.cc:1398] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.023s user 0.010s sys 0.001s
I20240501 14:01:44.245023 32584 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:44.252655 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-a733857ad51d452a9054cc5f533edc8e received from {username='slave'} at 127.22.5.197:39969
I20240501 14:01:44.252825 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-a733857ad51d452a9054cc5f533edc8e on tablet a733857ad51d452a9054cc5f533edc8e with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:44.258392 32558 ts_tablet_manager.cc:1398] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.039s user 0.007s sys 0.004s
I20240501 14:01:44.259840 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-c40a79b118424a5699bddd1c4ac4bb0e received from {username='slave'} at 127.22.5.199:33017
I20240501 14:01:44.260067 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-c40a79b118424a5699bddd1c4ac4bb0e on tablet c40a79b118424a5699bddd1c4ac4bb0e with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:44.271566 32584 log.cc:824] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:44.287925 32612 tablet_copy_client.cc:523] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.290866 32612 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:44.361650 32612 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.362442 32612 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:44.362951 32612 ts_tablet_manager.cc:1367] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.072s user 0.061s sys 0.004s
I20240501 14:01:44.364332 32612 raft_consensus.cc:348] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.365087 32612 raft_consensus.cc:729] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:44.365495 32612 consensus_queue.cc:260] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.366516 32612 ts_tablet_manager.cc:1398] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:44.367736 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-427af2a000f34ab6b9e3c9c7d7be1b32 received from {username='slave'} at 127.22.5.199:46389
I20240501 14:01:44.367875 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-427af2a000f34ab6b9e3c9c7d7be1b32 on tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:44.373606 32624 tablet_copy_client.cc:523] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.377656 32624 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:44.393987 32584 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.394673 32584 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:44.395193 32584 ts_tablet_manager.cc:1367] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.150s user 0.066s sys 0.008s
I20240501 14:01:44.407503 32584 raft_consensus.cc:348] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:44.408167 32584 raft_consensus.cc:729] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:44.408939 32584 consensus_queue.cc:260] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:44.417647 32584 ts_tablet_manager.cc:1398] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.022s user 0.013s sys 0.007s
I20240501 14:01:44.421978 32210 raft_consensus.cc:1203] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.425221 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-5c35ccca22fb45b78ee66991027c0261 received from {username='slave'} at 127.22.5.200:42889
I20240501 14:01:44.425388 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 744cef852ee9491db3928fa0092ef189-5c35ccca22fb45b78ee66991027c0261 on tablet 5c35ccca22fb45b78ee66991027c0261 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:44.435532 32634 tablet_copy_client.cc:523] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.436981 31946 raft_consensus.cc:1203] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.439570 32634 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:44.465409 32640 tablet_copy_client.cc:523] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.468465 32624 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.468927 32624 tablet_bootstrap.cc:492] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:44.469237 32624 ts_tablet_manager.cc:1367] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.092s user 0.052s sys 0.015s
I20240501 14:01:44.470299 32624 raft_consensus.cc:348] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:44.470536 32624 raft_consensus.cc:729] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:44.470762 32624 consensus_queue.cc:260] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:44.469103 32640 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:44.471580 32624 ts_tablet_manager.cc:1398] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:44.472647 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-4cc0201075e1466a9661b155c180d95f received from {username='slave'} at 127.22.5.197:45235
I20240501 14:01:44.472837 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-4cc0201075e1466a9661b155c180d95f on tablet 4cc0201075e1466a9661b155c180d95f with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:44.489620 32077 raft_consensus.cc:1203] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.509816 32634 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.510383 32634 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:44.510658 32634 ts_tablet_manager.cc:1367] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.071s user 0.036s sys 0.022s
I20240501 14:01:44.511842 32634 raft_consensus.cc:348] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:44.512101 32634 raft_consensus.cc:729] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:44.512307 32634 consensus_queue.cc:260] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:44.512923 32634 ts_tablet_manager.cc:1398] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:44.514276 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-642312cdad114fce839d2b60717bcc72 received from {username='slave'} at 127.22.5.200:39861
I20240501 14:01:44.514413 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session 744cef852ee9491db3928fa0092ef189-642312cdad114fce839d2b60717bcc72 on tablet 642312cdad114fce839d2b60717bcc72 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:44.535214 31813 raft_consensus.cc:1203] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.551857 32640 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.552371 32640 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:44.552660 32640 ts_tablet_manager.cc:1367] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.084s user 0.043s sys 0.009s
I20240501 14:01:44.553705 32640 raft_consensus.cc:348] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:44.553988 32640 raft_consensus.cc:729] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:44.554255 32640 consensus_queue.cc:260] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:44.555966 32640 ts_tablet_manager.cc:1398] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240501 14:01:44.558171 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-0ea9aaebd437424eb27c04bc2f6ffa7b received from {username='slave'} at 127.22.5.198:37341
I20240501 14:01:44.558315 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-0ea9aaebd437424eb27c04bc2f6ffa7b on tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:44.565456 32668 tablet_copy_client.cc:523] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.569296 32668 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:44.604261 32668 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:44.604699 32668 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:44.604995 32668 ts_tablet_manager.cc:1367] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.001s
I20240501 14:01:44.605727 32668 raft_consensus.cc:348] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.606029 32668 raft_consensus.cc:729] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:44.606243 32668 consensus_queue.cc:260] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:44.606931 32668 ts_tablet_manager.cc:1398] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 14:01:44.608049 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-f7d3626b1c8d458e9c6d4a28bc0564fa received from {username='slave'} at 127.22.5.199:33017
I20240501 14:01:44.608181 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-f7d3626b1c8d458e9c6d4a28bc0564fa on tablet f7d3626b1c8d458e9c6d4a28bc0564fa with peer 47b7351d2e9b4dbd9a09fd4133a97565
W20240501 14:01:44.692873 416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:44.693303 416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:44.693480 416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:44.705832 416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:44.705998 416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:44.706209 416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.201
I20240501 14:01:44.716707 416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.22.5.201
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.22.5.254:36941
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.201
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:44.718129 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:44.719895 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:01:44.724968 31813 raft_consensus.cc:1203] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:01:44.734112 469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:44.734349 470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:44.735116 472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:44.739745 416 server_base.cc:1034] running on GCE node
I20240501 14:01:44.740447 416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:44.741549 416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:44.742813 416 hybrid_clock.cc:648] HybridClock initialized: now 1714572104742728 us; error 54 us; skew 500 ppm
I20240501 14:01:44.743325 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:44.745939 416 webserver.cc:460] Webserver started at http://127.22.5.201:46283/ using document root <none> and password file <none>
I20240501 14:01:44.746846 416 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:44.746960 416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:44.747262 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:44.750270 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "aa11715897034babb1009134d8914066"
format_stamp: "Formatted at 2024-05-01 14:01:44 on dist-test-slave-5jzj"
I20240501 14:01:44.751526 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "aa11715897034babb1009134d8914066"
format_stamp: "Formatted at 2024-05-01 14:01:44 on dist-test-slave-5jzj"
I20240501 14:01:44.757567 416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 14:01:44.761896 478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:44.763490 416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:44.763650 416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "aa11715897034babb1009134d8914066"
format_stamp: "Formatted at 2024-05-01 14:01:44 on dist-test-slave-5jzj"
I20240501 14:01:44.763902 416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:44.782379 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:44.783432 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:44.783706 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:44.784821 416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:44.786839 416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:44.786931 416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:44.787060 416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:44.787144 416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:44.816944 32077 raft_consensus.cc:1203] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.848081 32513 raft_consensus.cc:1050] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:44.849195 32513 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:44.854669 31946 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.855471 30032 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.856060 32509 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.856930 32592 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.862344 30161 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.863057 32592 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.890362 31944 raft_consensus.cc:1203] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:44.896798 32592 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:44.901345 31946 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:44.905946 30031 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:44.910967 29753 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:44.914621 569 raft_consensus.cc:1050] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:44.915441 569 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:44.912353 30161 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:44.931988 32210 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.936805 32592 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.941776 30162 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.942133 30290 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:44.943094 569 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.943657 32592 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:44.951453 32665 raft_consensus.cc:1050] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:44.952747 32665 consensus_queue.cc:237] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:44.971894 32741 tablet_copy_client.cc:523] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.975771 32741 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:44.982153 32740 tablet_copy_client.cc:523] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.987555 32451 raft_consensus.cc:1050] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:44.988700 32740 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:44.988750 32451 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:44.990890 32750 tablet_copy_client.cc:523] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:44.996775 32750 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:45.007226 31813 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.008731 32550 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.014518 32077 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.014783 29901 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.015388 32592 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.020269 32550 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.020941 630 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.039099 30292 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.047941 29757 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.053756 32210 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.059967 30290 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.061060 32550 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.065665 32209 raft_consensus.cc:1203] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.078580 30161 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.079300 32550 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.095080 32550 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.095999 32076 raft_consensus.cc:1203] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.103363 630 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.104940 32076 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.111799 32767 tablet_copy_client.cc:523] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.113034 300 tablet_copy_client.cc:523] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.113148 30161 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.115429 32767 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:45.116611 300 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:45.118093 32750 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.118049 29760 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.118551 32750 tablet_bootstrap.cc:492] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:45.118814 32750 ts_tablet_manager.cc:1367] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.122s user 0.067s sys 0.007s
I20240501 14:01:45.119936 32750 raft_consensus.cc:348] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:45.120221 32750 raft_consensus.cc:729] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:45.120417 32750 consensus_queue.cc:260] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:45.121079 32750 ts_tablet_manager.cc:1398] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:45.121975 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-c953ab40905a4d23b9b9c970d25168b9 received from {username='slave'} at 127.22.5.198:37341
I20240501 14:01:45.122141 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-c953ab40905a4d23b9b9c970d25168b9 on tablet c953ab40905a4d23b9b9c970d25168b9 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:45.124410 30292 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.130587 31813 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.135638 29901 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.137514 30162 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.138157 629 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.144501 30161 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.149709 29899 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:45.151882 31946 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.152590 637 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.153738 30160 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.157855 32513 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.159636 32592 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.161732 29759 catalog_manager.cc:5484] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.164937 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 368edc181c894686a213fc542f328192 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.166406 31946 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.173967 30160 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.177809 30032 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:45.180191 30292 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.180495 32077 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.181010 631 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.181725 32665 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.183879 32665 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.189651 30292 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.195822 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.197582 32077 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.199693 29899 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:45.200732 30162 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
W20240501 14:01:45.201551 29744 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 368edc181c894686a213fc542f328192 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240501 14:01:45.202307 32210 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.203007 637 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.205778 32513 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.210451 32210 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.212452 29758 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:45.218266 29760 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.219133 30161 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.219779 32513 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.221509 29754 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.221891 30160 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.224486 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c35ccca22fb45b78ee66991027c0261 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.228796 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:45.236862 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c35ccca22fb45b78ee66991027c0261 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:45.237478 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 368edc181c894686a213fc542f328192 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:45.260479 656 tablet_replica.cc:330] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:45.261790 657 tablet_replica.cc:330] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:45.262435 657 raft_consensus.cc:2229] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.264140 657 raft_consensus.cc:2258] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.268720 656 raft_consensus.cc:2229] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.269217 656 raft_consensus.cc:2258] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.272531 656 ts_tablet_manager.cc:1875] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.274821 658 tablet_replica.cc:330] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:45.275444 657 ts_tablet_manager.cc:1875] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.276103 658 raft_consensus.cc:2229] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.277498 658 raft_consensus.cc:2258] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.278415 32741 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.280664 32723 raft_consensus.cc:1050] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:45.282006 32723 consensus_queue.cc:237] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:45.284041 658 ts_tablet_manager.cc:1875] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.285408 31813 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.286052 30292 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.286406 29899 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.286927 32730 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.287240 32568 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.287683 667 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.287286 300 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.300702 656 ts_tablet_manager.cc:1888] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.301040 656 log.cc:1196] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:01:45.303575 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 427af2a000f34ab6b9e3c9c7d7be1b32 (table rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]) successfully deleted
I20240501 14:01:45.330684 300 tablet_bootstrap.cc:492] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:45.331037 300 ts_tablet_manager.cc:1367] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.215s user 0.067s sys 0.014s
I20240501 14:01:45.331905 300 raft_consensus.cc:348] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.332187 300 raft_consensus.cc:729] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:45.332432 300 consensus_queue.cc:260] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.332990 300 ts_tablet_manager.cc:1398] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:45.339555 32741 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:45.339963 32741 ts_tablet_manager.cc:1367] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.364s user 0.059s sys 0.000s
I20240501 14:01:45.340345 32592 raft_consensus.cc:1050] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:45.341245 32592 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:45.344130 32740 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.344645 32740 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:45.344945 32740 ts_tablet_manager.cc:1367] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.356s user 0.050s sys 0.005s
I20240501 14:01:45.345758 32740 raft_consensus.cc:348] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.346029 32740 raft_consensus.cc:729] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:45.346239 32740 consensus_queue.cc:260] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.346629 338 tablet_copy_client.cc:523] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.346889 32740 ts_tablet_manager.cc:1398] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 14:01:45.347702 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-56f632889b1e400392fe710bf9f6f3ec received from {username='slave'} at 127.22.5.197:37661
I20240501 14:01:45.347829 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-56f632889b1e400392fe710bf9f6f3ec on tablet 56f632889b1e400392fe710bf9f6f3ec with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:45.352419 338 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:45.353672 321 tablet_copy_client.cc:523] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.334259 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-9735c5d9f784459ebafd8f2ff4f69763 received from {username='slave'} at 127.22.5.200:36943
I20240501 14:01:45.358175 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 744cef852ee9491db3928fa0092ef189-9735c5d9f784459ebafd8f2ff4f69763 on tablet 9735c5d9f784459ebafd8f2ff4f69763 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:45.358681 321 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:45.362695 31812 raft_consensus.cc:1203] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.340996 32741 raft_consensus.cc:348] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.365330 32741 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:45.365620 32741 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.366432 32741 ts_tablet_manager.cc:1398] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:01:45.367219 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-582bddb40ece4d5b8dcd3e1c00a53536 received from {username='slave'} at 127.22.5.200:42889
I20240501 14:01:45.367386 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 744cef852ee9491db3928fa0092ef189-582bddb40ece4d5b8dcd3e1c00a53536 on tablet 582bddb40ece4d5b8dcd3e1c00a53536 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:45.369045 31946 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.369316 30032 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.369648 32592 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.370141 32509 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.371043 30290 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.376250 32568 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.378003 30292 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.379446 324 tablet_copy_client.cc:523] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.381302 32509 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.382557 29899 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.384104 324 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:45.384073 29752 catalog_manager.cc:5484] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.384939 657 ts_tablet_manager.cc:1888] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.385107 657 log.cc:1196] T 5c35ccca22fb45b78ee66991027c0261 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c35ccca22fb45b78ee66991027c0261
I20240501 14:01:45.395686 31813 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.399518 30162 consensus_queue.cc:237] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:45.400858 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 5c35ccca22fb45b78ee66991027c0261 (table rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]) successfully deleted
I20240501 14:01:45.401883 658 ts_tablet_manager.cc:1888] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.402164 658 log.cc:1196] T 368edc181c894686a213fc542f328192 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/368edc181c894686a213fc542f328192
I20240501 14:01:45.402339 30290 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.403000 32730 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.403998 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 368edc181c894686a213fc542f328192 (table rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]) successfully deleted
I20240501 14:01:45.406602 32730 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.410763 30290 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.412570 32592 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.415323 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cc0201075e1466a9661b155c180d95f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.417762 30031 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.423136 31946 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.425388 29752 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:45.435232 29760 catalog_manager.cc:5484] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:45.438714 685 raft_consensus.cc:1050] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:45.439958 685 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:45.441396 344 tablet_copy_client.cc:523] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.445525 31812 raft_consensus.cc:1261] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.448449 344 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:45.448467 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc0201075e1466a9661b155c180d95f with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:45.450065 667 consensus_queue.cc:1035] T 4cc0201075e1466a9661b155c180d95f P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.452157 31812 raft_consensus.cc:2932] T 4cc0201075e1466a9661b155c180d95f P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.453753 688 tablet_replica.cc:330] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:45.454252 688 raft_consensus.cc:2229] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.454696 688 raft_consensus.cc:2258] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.459832 688 ts_tablet_manager.cc:1875] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.467150 688 ts_tablet_manager.cc:1888] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.474052 693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.201:38269 every 8 connection(s)
I20240501 14:01:45.475461 32210 raft_consensus.cc:1203] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.477984 688 log.cc:1196] T 4cc0201075e1466a9661b155c180d95f P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cc0201075e1466a9661b155c180d95f
I20240501 14:01:45.479071 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 4cc0201075e1466a9661b155c180d95f (table rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]) successfully deleted
I20240501 14:01:45.482828 30291 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.486263 29899 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:45.489360 31946 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.489466 30032 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.490098 32592 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.490633 32592 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.493059 32509 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.493549 31946 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.499083 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.500473 29756 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.502283 370 tablet_copy_client.cc:523] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:45.502475 30031 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.502624 32077 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.503058 687 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.505875 370 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:45.509351 687 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.513154 416 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.201:38269
I20240501 14:01:45.516644 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:45.517144 657 tablet_replica.cc:330] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:45.517383 657 raft_consensus.cc:2229] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.517609 657 raft_consensus.cc:2258] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.519469 416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 14:01:45.519672 657 ts_tablet_manager.cc:1875] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.520874 30162 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.521672 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 416
I20240501 14:01:45.521601 687 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.521910 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240501 14:01:45.529151 30032 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:45.535017 657 ts_tablet_manager.cc:1888] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.535183 657 log.cc:1196] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:01:45.538836 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 0ea9aaebd437424eb27c04bc2f6ffa7b (table rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]) successfully deleted
I20240501 14:01:45.541836 685 raft_consensus.cc:1050] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:45.551771 32604 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.552699 30162 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.560941 32077 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.562000 324 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.572120 30031 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.573424 685 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:45.573691 324 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:45.574127 324 ts_tablet_manager.cc:1367] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.190s user 0.054s sys 0.012s
I20240501 14:01:45.575047 324 raft_consensus.cc:348] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.575429 324 raft_consensus.cc:729] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:45.575680 324 consensus_queue.cc:260] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.577693 324 ts_tablet_manager.cc:1398] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:45.578547 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-212e139f1e7c4674987c2309efe3e495 received from {username='slave'} at 127.22.5.197:38005
I20240501 14:01:45.578711 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-212e139f1e7c4674987c2309efe3e495 on tablet 212e139f1e7c4674987c2309efe3e495 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:45.568017 29752 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.594106 32077 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.595131 701 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.599032 30162 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.599730 32604 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.599925 29899 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.600833 701 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.608651 30290 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:45.611800 32077 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.612571 701 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.615936 30032 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.616047 701 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.616904 32604 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.618479 30032 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.630411 29754 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.632977 31946 raft_consensus.cc:1203] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.636468 344 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.637102 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet c40a79b118424a5699bddd1c4ac4bb0e with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:45.637523 656 tablet_replica.cc:330] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:45.637737 656 raft_consensus.cc:2229] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.637984 656 raft_consensus.cc:2258] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.638705 344 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:45.638998 656 ts_tablet_manager.cc:1875] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.639081 344 ts_tablet_manager.cc:1367] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.194s user 0.049s sys 0.004s
I20240501 14:01:45.639843 344 raft_consensus.cc:348] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:45.640142 344 raft_consensus.cc:729] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:45.640365 344 consensus_queue.cc:260] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:45.641019 344 ts_tablet_manager.cc:1398] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:45.641731 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-98d94ed11f0f4a9d9f519a24c3320ce4 received from {username='slave'} at 127.22.5.198:60129
I20240501 14:01:45.641865 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-98d94ed11f0f4a9d9f519a24c3320ce4 on tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:45.642378 32767 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.643035 32730 raft_consensus.cc:1050] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:45.643882 32730 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:45.647220 656 ts_tablet_manager.cc:1888] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.647359 656 log.cc:1196] T c40a79b118424a5699bddd1c4ac4bb0e P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:01:45.648183 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet c40a79b118424a5699bddd1c4ac4bb0e (table rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]) successfully deleted
I20240501 14:01:45.647219 702 heartbeater.cc:347] Connected to a master server at 127.22.5.254:36941
I20240501 14:01:45.649915 702 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:45.651211 702 heartbeater.cc:511] Master 127.22.5.254:36941 requested a full tablet report, sending...
I20240501 14:01:45.651093 370 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.652824 370 tablet_bootstrap.cc:492] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:45.652791 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c40a79b118424a5699bddd1c4ac4bb0e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.656900 29754 ts_manager.cc:194] Registered new tserver with Master: aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:01:45.658560 29754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.201:36119
I20240501 14:01:45.658892 32767 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:45.659263 32767 ts_tablet_manager.cc:1367] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.544s user 0.054s sys 0.000s
I20240501 14:01:45.660040 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.660705 32568 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.661259 685 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.663448 32075 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.666201 32767 raft_consensus.cc:348] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:45.666456 32767 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:45.666353 29899 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.666662 32767 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:45.668979 32767 ts_tablet_manager.cc:1398] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:01:45.671586 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-ef202920b8d64287a8a2cdb40c73a819 received from {username='slave'} at 127.22.5.199:57317
I20240501 14:01:45.671767 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-ef202920b8d64287a8a2cdb40c73a819 on tablet ef202920b8d64287a8a2cdb40c73a819 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:45.673027 32076 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.683063 29756 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.672796 30162 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.692011 30291 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:45.694195 32076 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.695513 700 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.697908 32210 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.698060 701 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.698769 32076 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.700909 30031 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.701601 32730 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.704699 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7d3626b1c8d458e9c6d4a28bc0564fa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.705760 29752 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.713686 32730 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.715091 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3626b1c8d458e9c6d4a28bc0564fa with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:45.715503 688 tablet_replica.cc:330] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:45.715710 688 raft_consensus.cc:2229] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.715986 688 raft_consensus.cc:2258] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.717077 688 ts_tablet_manager.cc:1875] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.723793 338 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.724346 338 tablet_bootstrap.cc:492] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:45.724815 338 ts_tablet_manager.cc:1367] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.373s user 0.055s sys 0.000s
I20240501 14:01:45.725672 338 raft_consensus.cc:348] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.726028 338 raft_consensus.cc:729] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:45.726290 338 consensus_queue.cc:260] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:45.726954 338 ts_tablet_manager.cc:1398] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:45.727814 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-d1bde99cadd94efc826b4074b2674104 received from {username='slave'} at 127.22.5.197:39969
I20240501 14:01:45.727950 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-d1bde99cadd94efc826b4074b2674104 on tablet d1bde99cadd94efc826b4074b2674104 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:45.728087 688 ts_tablet_manager.cc:1888] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.728212 688 log.cc:1196] T f7d3626b1c8d458e9c6d4a28bc0564fa P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:01:45.732758 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet f7d3626b1c8d458e9c6d4a28bc0564fa (table rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]) successfully deleted
I20240501 14:01:45.732789 32076 raft_consensus.cc:1203] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.741300 321 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:45.741514 30160 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.741904 321 tablet_bootstrap.cc:492] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:45.742219 321 ts_tablet_manager.cc:1367] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.384s user 0.062s sys 0.018s
I20240501 14:01:45.742537 30031 consensus_queue.cc:237] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:45.742794 701 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.743113 321 raft_consensus.cc:348] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.743350 321 raft_consensus.cc:729] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:45.743552 321 consensus_queue.cc:260] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:45.745188 321 ts_tablet_manager.cc:1398] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:45.744851 30160 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:45.745961 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-edf4bcf02c7e4352bd998e139802db62 received from {username='slave'} at 127.22.5.200:49589
I20240501 14:01:45.746099 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 744cef852ee9491db3928fa0092ef189-edf4bcf02c7e4352bd998e139802db62 on tablet edf4bcf02c7e4352bd998e139802db62 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:45.746901 31812 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.747946 629 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.748149 30161 raft_consensus.cc:1261] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.749940 370 ts_tablet_manager.cc:1367] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.147s user 0.051s sys 0.002s
I20240501 14:01:45.751492 629 consensus_queue.cc:1035] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.752447 370 raft_consensus.cc:348] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:45.752735 370 raft_consensus.cc:729] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:45.752614 629 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.752974 370 consensus_queue.cc:260] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:45.754320 370 ts_tablet_manager.cc:1398] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:01:45.755053 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-6241869f8e7f4bfaa65508fd61f405a6 received from {username='slave'} at 127.22.5.199:45965
I20240501 14:01:45.755185 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-6241869f8e7f4bfaa65508fd61f405a6 on tablet 6241869f8e7f4bfaa65508fd61f405a6 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:45.757647 30160 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.757810 31812 raft_consensus.cc:2932] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.763614 29754 catalog_manager.cc:5484] T a733857ad51d452a9054cc5f533edc8e P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.767693 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet a733857ad51d452a9054cc5f533edc8e with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:45.768074 688 tablet_replica.cc:330] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:45.768289 688 raft_consensus.cc:2229] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.768512 688 raft_consensus.cc:2258] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.769484 688 ts_tablet_manager.cc:1875] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.772634 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a733857ad51d452a9054cc5f533edc8e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.774436 32568 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.777279 688 ts_tablet_manager.cc:1888] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.777518 688 log.cc:1196] T a733857ad51d452a9054cc5f533edc8e P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/a733857ad51d452a9054cc5f533edc8e
I20240501 14:01:45.778503 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet a733857ad51d452a9054cc5f533edc8e (table rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]) successfully deleted
I20240501 14:01:45.778260 32209 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.784888 29752 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:45.787456 30290 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.786644 30032 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.791385 30160 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:45.794482 32209 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.795322 32730 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.800230 32730 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.804564 32210 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.805716 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 642312cdad114fce839d2b60717bcc72 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.807118 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.807838 32730 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.809616 30290 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:45.811836 29752 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:45.816725 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 642312cdad114fce839d2b60717bcc72 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:45.817170 658 tablet_replica.cc:330] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:45.817412 658 raft_consensus.cc:2229] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.817704 658 raft_consensus.cc:2258] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.826563 658 ts_tablet_manager.cc:1875] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.832482 658 ts_tablet_manager.cc:1888] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.832744 658 log.cc:1196] T 642312cdad114fce839d2b60717bcc72 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/642312cdad114fce839d2b60717bcc72
I20240501 14:01:45.833631 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 642312cdad114fce839d2b60717bcc72 (table rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]) successfully deleted
I20240501 14:01:45.885524 32210 raft_consensus.cc:1203] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.892633 32210 raft_consensus.cc:1203] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:45.918473 700 raft_consensus.cc:1050] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:45.919384 700 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:45.922284 31812 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.922359 29899 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.922642 30160 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:45.922966 32604 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.923456 685 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.923817 32604 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:45.930320 685 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.931564 30160 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.931639 31812 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.938576 29752 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.940950 29899 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.945842 30290 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:45.948073 29899 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.948850 31812 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:45.948876 701 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.951722 685 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:45.954130 687 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.954433 29899 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.958643 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56f632889b1e400392fe710bf9f6f3ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:45.958402 31812 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:45.960793 29754 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:45.966404 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f632889b1e400392fe710bf9f6f3ec with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:45.966822 656 tablet_replica.cc:330] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:45.967026 656 raft_consensus.cc:2229] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:45.967342 656 raft_consensus.cc:2258] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:45.968269 656 ts_tablet_manager.cc:1875] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:45.974416 30032 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:45.974881 656 ts_tablet_manager.cc:1888] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:45.975021 656 log.cc:1196] T 56f632889b1e400392fe710bf9f6f3ec P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/56f632889b1e400392fe710bf9f6f3ec
I20240501 14:01:45.975837 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 56f632889b1e400392fe710bf9f6f3ec (table rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]) successfully deleted
I20240501 14:01:45.976888 29899 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:45.977624 30160 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:45.978193 32665 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:45.979022 32665 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:45.982069 717 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:45.982805 30160 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:45.984256 29901 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:45.989941 29756 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:45.992780 29968 consensus_peers.cc:485] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0929994d486d4610b9cc76abd39f043a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.005416 31812 raft_consensus.cc:1203] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:46.005416 29899 consensus_queue.cc:237] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:46.008054 30032 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.008136 30290 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.008702 32509 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.009034 32592 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:46.009434 29836 consensus_peers.cc:485] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ccf3fc5fc7f44a3bc53dad7902c2d3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.012012 32509 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.012930 30290 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.013422 30032 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.015188 32592 raft_consensus.cc:1050] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:46.016070 32592 consensus_queue.cc:237] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:46.022668 29754 catalog_manager.cc:5484] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.027874 31946 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.028946 30160 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.028935 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:01:46.030349 32509 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.032263 629 raft_consensus.cc:1050] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:46.033278 629 consensus_queue.cc:237] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:46.039786 30290 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.040526 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.053748 29899 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.054607 716 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.056192 788 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.056587 32210 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.057711 30162 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.060264 30160 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.060395 30291 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.064837 629 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.065019 29754 catalog_manager.cc:5484] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.065310 716 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.072469 629 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.073714 30160 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.077895 29899 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.079120 29754 catalog_manager.cc:5484] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.083493 31944 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.087309 29901 consensus_queue.cc:237] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:46.090080 31945 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.091130 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.093379 30162 raft_consensus.cc:1261] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.094077 32592 consensus_queue.cc:1035] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.095126 32513 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.095971 31945 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.101200 32210 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.101526 29752 catalog_manager.cc:5484] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.102732 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c953ab40905a4d23b9b9c970d25168b9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.105926 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet c953ab40905a4d23b9b9c970d25168b9 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:46.109150 30032 consensus_queue.cc:237] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:46.111215 32210 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.111586 30162 raft_consensus.cc:1261] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.111703 32665 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.112695 32665 consensus_queue.cc:1035] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.113613 30160 raft_consensus.cc:2932] T c953ab40905a4d23b9b9c970d25168b9 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.116425 716 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.117378 30162 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.118326 807 tablet_replica.cc:330] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:46.118531 807 raft_consensus.cc:2229] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.118865 807 raft_consensus.cc:2258] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.120777 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9735c5d9f784459ebafd8f2ff4f69763 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.122382 29752 catalog_manager.cc:5484] T 9735c5d9f784459ebafd8f2ff4f69763 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.124433 807 ts_tablet_manager.cc:1875] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.126139 32210 raft_consensus.cc:2932] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.127117 32076 raft_consensus.cc:1203] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:46.127104 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9735c5d9f784459ebafd8f2ff4f69763 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:46.127486 688 tablet_replica.cc:330] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:46.127694 688 raft_consensus.cc:2229] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.127929 688 raft_consensus.cc:2258] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.129316 688 ts_tablet_manager.cc:1875] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.136495 807 ts_tablet_manager.cc:1888] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.136636 807 log.cc:1196] T c953ab40905a4d23b9b9c970d25168b9 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/c953ab40905a4d23b9b9c970d25168b9
I20240501 14:01:46.137478 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet c953ab40905a4d23b9b9c970d25168b9 (table rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]) successfully deleted
I20240501 14:01:46.138351 688 ts_tablet_manager.cc:1888] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.138579 688 log.cc:1196] T 9735c5d9f784459ebafd8f2ff4f69763 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/9735c5d9f784459ebafd8f2ff4f69763
I20240501 14:01:46.139515 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 9735c5d9f784459ebafd8f2ff4f69763 (table rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]) successfully deleted
I20240501 14:01:46.142431 31945 raft_consensus.cc:1203] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:46.153504 31812 raft_consensus.cc:1203] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:46.199072 32592 raft_consensus.cc:1050] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:46.199983 32592 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:46.206458 32076 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.207145 32509 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.207835 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.208451 32509 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.212218 30160 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.213019 32513 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.213585 32509 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.218667 30032 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.223914 32076 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.230871 29754 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.236181 29899 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:46.235999 30162 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.239342 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.239810 32076 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.240332 32509 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.240710 32513 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.244724 32513 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.250393 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef202920b8d64287a8a2cdb40c73a819 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.250967 32076 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.251610 30032 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.255264 29760 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.260814 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet ef202920b8d64287a8a2cdb40c73a819 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:46.261181 656 tablet_replica.cc:330] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:46.261402 656 raft_consensus.cc:2229] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.261638 656 raft_consensus.cc:2258] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.262516 656 ts_tablet_manager.cc:1875] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.269552 656 ts_tablet_manager.cc:1888] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.269692 656 log.cc:1196] T ef202920b8d64287a8a2cdb40c73a819 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/ef202920b8d64287a8a2cdb40c73a819
I20240501 14:01:46.270845 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet ef202920b8d64287a8a2cdb40c73a819 (table rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]) successfully deleted
I20240501 14:01:46.290418 32592 raft_consensus.cc:1050] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:46.291234 32592 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:46.295629 32210 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.295794 30032 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.295919 30291 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.296468 32513 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.296808 32592 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.297397 32509 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.301893 788 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.302634 30291 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.305842 30032 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.306695 32210 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.316289 29756 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.322942 29899 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:46.325598 32209 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.326638 788 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.326954 30291 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.327495 788 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.330529 32592 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.331480 30290 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.331610 32209 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.334839 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 582bddb40ece4d5b8dcd3e1c00a53536 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.338794 29760 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.344808 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 582bddb40ece4d5b8dcd3e1c00a53536 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:46.345541 862 tablet_replica.cc:330] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:46.345757 862 raft_consensus.cc:2229] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.346168 862 raft_consensus.cc:2258] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.347263 862 ts_tablet_manager.cc:1875] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.354477 862 ts_tablet_manager.cc:1888] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.354698 862 log.cc:1196] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:01:46.355540 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 582bddb40ece4d5b8dcd3e1c00a53536 (table rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]) successfully deleted
I20240501 14:01:46.369805 868 ts_tablet_manager.cc:902] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:46.374215 868 tablet_copy_client.cc:308] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:46.387568 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 0929994d486d4610b9cc76abd39f043a from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:45035)
I20240501 14:01:46.387773 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 0929994d486d4610b9cc76abd39f043a from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:45035: session id = 3321c47e58ba49f09f4f036ac72fc133-0929994d486d4610b9cc76abd39f043a
I20240501 14:01:46.390945 30052 tablet_copy_source_session.cc:205] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:46.392377 868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929994d486d4610b9cc76abd39f043a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:46.398778 868 tablet_copy_client.cc:783] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:46.399055 868 tablet_copy_client.cc:647] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:46.400592 872 tablet_copy_client.cc:1002] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:46.432755 687 raft_consensus.cc:1050] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:46.433845 687 consensus_queue.cc:237] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:46.436710 30032 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.437378 687 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.437669 30162 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.437758 32209 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.438536 32604 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.439033 32604 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.457517 32604 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.458714 30162 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.464638 30032 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.468991 908 ts_tablet_manager.cc:902] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:46.469347 29760 catalog_manager.cc:5484] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.476087 908 tablet_copy_client.cc:308] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:46.476866 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:57317)
I20240501 14:01:46.477047 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:57317: session id = 47b7351d2e9b4dbd9a09fd4133a97565-1ccf3fc5fc7f44a3bc53dad7902c2d3a
I20240501 14:01:46.480280 29921 tablet_copy_source_session.cc:205] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:46.480762 32209 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:46.481685 908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:46.488004 908 tablet_copy_client.cc:783] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:46.488212 908 tablet_copy_client.cc:647] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:46.489795 917 tablet_copy_client.cc:1002] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:46.498132 30290 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:46.500509 30162 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.500552 30032 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.500676 30228 consensus_peers.cc:485] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b56df9dcda34e6a8d27a8a6b24742b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.501214 32604 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.501586 687 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.504019 685 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.504786 30162 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.504740 30031 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.512208 29756 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.527051 30031 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:46.529292 29899 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.529562 29965 consensus_peers.cc:485] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 594dcb0a05054f2abaa11e5d4537b724. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.529654 30290 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.529902 629 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.530200 32665 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.533231 32665 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.533984 30290 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.534526 29899 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.539332 29760 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.555060 29899 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:46.557490 30290 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.557631 29837 consensus_peers.cc:485] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31303ad8c9124217a45e26f8da5cc0a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.557731 30031 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.558187 32592 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.558460 32509 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.561398 32513 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.562100 30290 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.567749 29756 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.571729 30032 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.578380 716 raft_consensus.cc:1050] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:46.579244 716 consensus_queue.cc:237] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:46.583271 31812 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.583583 29899 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.584237 629 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.584852 30162 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.585002 717 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.585510 629 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.588613 32509 raft_consensus.cc:1050] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:46.589411 32509 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:46.590970 629 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.596429 29899 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.602027 31813 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.601745 31812 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.602661 30162 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.603035 788 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.605249 32513 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.606163 30162 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.608404 29752 catalog_manager.cc:5484] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.617973 30032 consensus_queue.cc:237] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:46.622189 30032 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.622372 31813 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.622807 32592 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.623310 629 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.625278 29899 raft_consensus.cc:1261] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.625804 629 consensus_queue.cc:1035] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.627063 32730 raft_consensus.cc:1050] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:46.627920 32730 consensus_queue.cc:237] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:46.628199 716 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.632120 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1bde99cadd94efc826b4074b2674104 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.634232 29901 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.639905 29756 catalog_manager.cc:5484] T d1bde99cadd94efc826b4074b2674104 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.641589 31813 raft_consensus.cc:2932] T d1bde99cadd94efc826b4074b2674104 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.647233 32076 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.647812 32723 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.655332 30290 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.656400 32723 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.663719 702 heartbeater.cc:503] Master 127.22.5.254:36941 was elected leader, sending a full tablet report...
I20240501 14:01:46.666721 30032 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.671273 957 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.674017 32509 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.679610 31813 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.680385 30160 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.682482 30031 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.693171 962 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.695240 667 raft_consensus.cc:1050] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:46.696314 667 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:46.704624 30031 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:46.705411 31945 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.706014 29899 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.707134 962 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.707809 962 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.708258 29760 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.712339 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet d1bde99cadd94efc826b4074b2674104 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:46.712719 656 tablet_replica.cc:330] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:46.712932 656 raft_consensus.cc:2229] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.713239 656 raft_consensus.cc:2258] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:46.714926 30097 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:40447, user_credentials={real_user=slave}} blocked reactor thread for 43804.7us
I20240501 14:01:46.715183 656 ts_tablet_manager.cc:1875] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.718951 30032 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.719353 30290 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.722246 29901 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:46.724521 30290 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:46.724697 30160 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.725155 31813 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.725344 32513 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.725924 32513 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.726608 29756 catalog_manager.cc:5484] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:46.726877 667 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:46.729616 32592 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.730418 30160 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.736136 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 212e139f1e7c4674987c2309efe3e495 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.736443 29754 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.739446 31813 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:46.741650 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 212e139f1e7c4674987c2309efe3e495 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:46.742079 862 tablet_replica.cc:330] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:46.742352 862 raft_consensus.cc:2229] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.742784 862 raft_consensus.cc:2258] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.744494 862 ts_tablet_manager.cc:1875] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.745488 656 ts_tablet_manager.cc:1888] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.745625 656 log.cc:1196] T d1bde99cadd94efc826b4074b2674104 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1bde99cadd94efc826b4074b2674104
I20240501 14:01:46.746596 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet d1bde99cadd94efc826b4074b2674104 (table rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]) successfully deleted
I20240501 14:01:46.747761 30291 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.748293 629 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.749017 30160 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.749580 629 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.753862 862 ts_tablet_manager.cc:1888] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.754096 862 log.cc:1196] T 212e139f1e7c4674987c2309efe3e495 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/212e139f1e7c4674987c2309efe3e495
I20240501 14:01:46.755103 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 212e139f1e7c4674987c2309efe3e495 (table rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]) successfully deleted
W20240501 14:01:46.758636 29968 consensus_peers.cc:485] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4988f0de3214860a65007b8e31e88a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.761754 667 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.766258 29901 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.768016 31945 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.769804 29760 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.775923 30290 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.776609 629 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:46.779873 30160 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:46.781016 30161 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:46.785547 30290 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.786142 667 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.789028 667 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.790699 30292 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.794585 29760 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.805533 29752 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.812040 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:46.813501 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:46.815930 31946 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:46.817785 667 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:46.821451 31945 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:46.831794 30290 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:46.838037 973 tablet_replica.cc:330] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:46.838385 973 raft_consensus.cc:2229] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:46.838735 973 raft_consensus.cc:2258] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:46.842170 973 ts_tablet_manager.cc:1875] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:46.848523 973 ts_tablet_manager.cc:1888] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:46.848718 973 log.cc:1196] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:01:46.850212 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 98d94ed11f0f4a9d9f519a24c3320ce4 (table rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]) successfully deleted
I20240501 14:01:46.861794 30031 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:46.864281 29901 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.864332 30290 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.864996 717 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:46.865355 29967 consensus_peers.cc:485] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1ada0fe1ce431a823f2814f116db28. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.865636 717 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.868146 629 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.868986 29901 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.869066 30291 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.875144 29756 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:46.898703 30162 consensus_queue.cc:237] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:46.901453 29901 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.901647 30095 consensus_peers.cc:485] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48b5a3abfd8348b9ae47c8867642044e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.901726 30291 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.902145 961 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.902501 32730 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.905299 961 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.906013 29901 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.906411 30291 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.911176 29754 catalog_manager.cc:5484] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.926823 30290 consensus_queue.cc:237] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:46.929337 30031 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.929504 30228 consensus_peers.cc:485] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64012de915564c7887e44d432b170fdc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.929533 30162 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.929833 685 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.930135 687 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.933284 687 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.933796 30031 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.933773 30162 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.938771 29756 catalog_manager.cc:5484] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index -1 to 4, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:46.954839 30031 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:46.957680 29901 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:46.957720 29965 consensus_peers.cc:485] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c00649352f9d41c3a12a5e767b8b2aa6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.957917 30162 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:46.958307 716 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.958601 717 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.961711 717 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.962227 30162 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.963274 29901 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:46.969326 29756 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index -1 to 4, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:47.020228 1031 ts_tablet_manager.cc:902] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:47.023108 1031 tablet_copy_client.cc:308] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:47.023834 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 594dcb0a05054f2abaa11e5d4537b724 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:39969)
I20240501 14:01:47.024029 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 594dcb0a05054f2abaa11e5d4537b724 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:39969: session id = be55dad80e3a4ad79d21aaa3199b1d0b-594dcb0a05054f2abaa11e5d4537b724
I20240501 14:01:47.027113 30052 tablet_copy_source_session.cc:205] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.028610 1031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594dcb0a05054f2abaa11e5d4537b724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.034447 1031 tablet_copy_client.cc:783] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.034781 1031 tablet_copy_client.cc:647] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.035238 1041 ts_tablet_manager.cc:902] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:47.036761 1035 tablet_copy_client.cc:1002] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.041862 1041 tablet_copy_client.cc:308] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:47.042809 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 31303ad8c9124217a45e26f8da5cc0a1 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:42889)
I20240501 14:01:47.043056 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 31303ad8c9124217a45e26f8da5cc0a1 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:42889: session id = 744cef852ee9491db3928fa0092ef189-31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:01:47.045655 29921 tablet_copy_source_session.cc:205] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.047091 1041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31303ad8c9124217a45e26f8da5cc0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.053516 1041 tablet_copy_client.cc:783] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.053771 1041 tablet_copy_client.cc:647] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.056516 1050 tablet_copy_client.cc:1002] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.082360 32076 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:47.086953 30162 consensus_queue.cc:237] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:47.089180 32076 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:47.089239 30290 raft_consensus.cc:1261] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:47.089926 667 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:47.090270 962 consensus_queue.cc:1035] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:47.093005 32730 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:47.097699 30290 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:47.097708 32075 raft_consensus.cc:2932] T 6241869f8e7f4bfaa65508fd61f405a6 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:47.098584 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6241869f8e7f4bfaa65508fd61f405a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:47.101765 29752 catalog_manager.cc:5484] T 6241869f8e7f4bfaa65508fd61f405a6 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.106300 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 6241869f8e7f4bfaa65508fd61f405a6 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:47.106710 862 tablet_replica.cc:330] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:47.106992 862 raft_consensus.cc:2229] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:47.107281 862 raft_consensus.cc:2258] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:47.108300 862 ts_tablet_manager.cc:1875] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:47.115495 862 ts_tablet_manager.cc:1888] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:47.115665 862 log.cc:1196] T 6241869f8e7f4bfaa65508fd61f405a6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/6241869f8e7f4bfaa65508fd61f405a6
I20240501 14:01:47.116571 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 6241869f8e7f4bfaa65508fd61f405a6 (table rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]) successfully deleted
I20240501 14:01:47.117524 1077 ts_tablet_manager.cc:902] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:47.119190 1077 tablet_copy_client.cc:308] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:47.120018 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:49589)
I20240501 14:01:47.120236 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet 9b56df9dcda34e6a8d27a8a6b24742b2 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:49589: session id = 744cef852ee9491db3928fa0092ef189-9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:01:47.123126 30312 tablet_copy_source_session.cc:205] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.124423 1077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56df9dcda34e6a8d27a8a6b24742b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.126827 30290 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62"
dest_uuid: "4a15e881772b41eab8a71fc027390db9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41088
I20240501 14:01:47.127130 30290 raft_consensus.cc:595] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:47.131074 1077 tablet_copy_client.cc:783] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.131356 1077 tablet_copy_client.cc:647] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.133344 1082 tablet_copy_client.cc:1002] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.171320 1106 ts_tablet_manager.cc:902] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:47.174413 1106 tablet_copy_client.cc:308] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:47.175223 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet f4988f0de3214860a65007b8e31e88a5 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:45035)
I20240501 14:01:47.175401 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet f4988f0de3214860a65007b8e31e88a5 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:45035: session id = 3321c47e58ba49f09f4f036ac72fc133-f4988f0de3214860a65007b8e31e88a5
I20240501 14:01:47.178918 30052 tablet_copy_source_session.cc:205] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.180337 1106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4988f0de3214860a65007b8e31e88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.185530 1106 tablet_copy_client.cc:783] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.185878 1106 tablet_copy_client.cc:647] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.187491 1111 tablet_copy_client.cc:1002] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.293644 1157 ts_tablet_manager.cc:902] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:47.295226 1157 tablet_copy_client.cc:308] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:47.295974 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 48b5a3abfd8348b9ae47c8867642044e from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:45235)
I20240501 14:01:47.296165 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 48b5a3abfd8348b9ae47c8867642044e from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:45235: session id = be55dad80e3a4ad79d21aaa3199b1d0b-48b5a3abfd8348b9ae47c8867642044e
I20240501 14:01:47.298590 30182 tablet_copy_source_session.cc:205] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.299827 1157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b5a3abfd8348b9ae47c8867642044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.304417 30290 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62"
dest_uuid: "4a15e881772b41eab8a71fc027390db9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41108
I20240501 14:01:47.304654 30290 raft_consensus.cc:595] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:47.304808 1157 tablet_copy_client.cc:783] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.305096 1157 tablet_copy_client.cc:647] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.307639 1161 tablet_copy_client.cc:1002] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.387929 687 raft_consensus.cc:979] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: : Instructing follower 76c7a691de8a4e31b32799f43cb4b812 to start an election
I20240501 14:01:47.388108 687 raft_consensus.cc:1067] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Signalling peer 76c7a691de8a4e31b32799f43cb4b812 to start an election
I20240501 14:01:47.389240 30162 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62"
dest_uuid: "76c7a691de8a4e31b32799f43cb4b812"
from {username='slave'} at 127.22.5.196:43691
I20240501 14:01:47.389930 30162 raft_consensus.cc:482] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.390070 30162 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.392850 30162 raft_consensus.cc:504] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:47.394145 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:47.394204 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:47.394407 30031 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.394409 30290 raft_consensus.cc:3032] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.394502 30290 raft_consensus.cc:729] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:47.394877 30290 consensus_queue.cc:260] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:47.395304 30290 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.397330 30031 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 2.
I20240501 14:01:47.397927 30290 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 2.
I20240501 14:01:47.398548 30097 leader_election.cc:302] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:47.398886 959 raft_consensus.cc:2781] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.399055 959 raft_consensus.cc:686] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 2 LEADER]: Becoming Leader. State: Replica: 76c7a691de8a4e31b32799f43cb4b812, State: Running, Role: LEADER
I20240501 14:01:47.399405 959 consensus_queue.cc:237] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:47.399294 30162 leader_election.cc:288] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [CANDIDATE]: Term 2 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:01:47.400477 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf4bcf02c7e4352bd998e139802db62" candidate_uuid: "76c7a691de8a4e31b32799f43cb4b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "744cef852ee9491db3928fa0092ef189"
I20240501 14:01:47.400822 32209 raft_consensus.cc:3037] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.402714 29752 catalog_manager.cc:5484] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: term changed from 1 to 2, leader changed from 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) to 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195). New cstate: current_term: 2 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.403674 32209 raft_consensus.cc:2454] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c7a691de8a4e31b32799f43cb4b812 in term 2.
I20240501 14:01:47.430835 1212 ts_tablet_manager.cc:902] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:47.432473 1212 tablet_copy_client.cc:308] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:47.433282 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet c00649352f9d41c3a12a5e767b8b2aa6 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:39969)
I20240501 14:01:47.433449 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet c00649352f9d41c3a12a5e767b8b2aa6 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:39969: session id = be55dad80e3a4ad79d21aaa3199b1d0b-c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:01:47.435636 30052 tablet_copy_source_session.cc:205] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.437067 1212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00649352f9d41c3a12a5e767b8b2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.441090 1219 ts_tablet_manager.cc:902] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:47.443327 1212 tablet_copy_client.cc:783] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.443579 1212 tablet_copy_client.cc:647] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.445101 1218 tablet_copy_client.cc:1002] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.446861 1224 ts_tablet_manager.cc:902] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:47.449337 1219 tablet_copy_client.cc:308] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:47.450485 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet 64012de915564c7887e44d432b170fdc from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:49589)
I20240501 14:01:47.450762 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet 64012de915564c7887e44d432b170fdc from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:49589: session id = 744cef852ee9491db3928fa0092ef189-64012de915564c7887e44d432b170fdc
I20240501 14:01:47.454706 30312 tablet_copy_source_session.cc:205] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.456363 1219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64012de915564c7887e44d432b170fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.461328 1224 tablet_copy_client.cc:308] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:47.462246 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 2b1ada0fe1ce431a823f2814f116db28 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:46389)
I20240501 14:01:47.462280 1219 tablet_copy_client.cc:783] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.462424 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 2b1ada0fe1ce431a823f2814f116db28 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:46389: session id = 47b7351d2e9b4dbd9a09fd4133a97565-2b1ada0fe1ce431a823f2814f116db28
I20240501 14:01:47.462541 1219 tablet_copy_client.cc:647] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.464298 1227 tablet_copy_client.cc:1002] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.465204 30052 tablet_copy_source_session.cc:205] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.467034 1224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1ada0fe1ce431a823f2814f116db28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.473415 1224 tablet_copy_client.cc:783] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.473769 1224 tablet_copy_client.cc:647] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.475610 1231 tablet_copy_client.cc:1002] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:47.810096 32209 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:47.810712 32730 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:47.890158 30031 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:47.890784 32730 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:47.895375 30290 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:47.896104 961 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:47.898905 30162 consensus_queue.cc:237] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:47.901017 868 tablet_copy_client.cc:523] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.905522 30032 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:47.905927 868 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:47.907850 32730 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:47.915536 961 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:47.918373 32210 raft_consensus.cc:1261] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:47.919605 30031 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:47.920179 32730 consensus_queue.cc:1035] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:47.921727 32210 raft_consensus.cc:2932] T edf4bcf02c7e4352bd998e139802db62 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:47.922165 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edf4bcf02c7e4352bd998e139802db62 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:47.927174 29754 catalog_manager.cc:5484] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 7, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 2 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.932792 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet edf4bcf02c7e4352bd998e139802db62 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:47.936443 1315 tablet_replica.cc:330] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:47.936654 1315 raft_consensus.cc:2229] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:47.936890 1315 raft_consensus.cc:2258] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:47.937835 1315 ts_tablet_manager.cc:1875] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:47.943487 1315 ts_tablet_manager.cc:1888] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:01:47.943610 1315 log.cc:1196] T edf4bcf02c7e4352bd998e139802db62 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/edf4bcf02c7e4352bd998e139802db62
I20240501 14:01:47.944389 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet edf4bcf02c7e4352bd998e139802db62 (table rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]) successfully deleted
I20240501 14:01:47.960300 868 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:47.960850 868 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:47.961187 868 ts_tablet_manager.cc:1367] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.055s user 0.038s sys 0.010s
I20240501 14:01:47.962116 868 raft_consensus.cc:348] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:47.962424 868 raft_consensus.cc:729] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:47.962669 868 consensus_queue.cc:260] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:47.963460 868 ts_tablet_manager.cc:1398] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:47.964424 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-0929994d486d4610b9cc76abd39f043a received from {username='slave'} at 127.22.5.198:45035
I20240501 14:01:47.964562 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-0929994d486d4610b9cc76abd39f043a on tablet 0929994d486d4610b9cc76abd39f043a with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:47.990172 908 tablet_copy_client.cc:523] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.993109 908 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:48.032248 908 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.032649 908 tablet_bootstrap.cc:492] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:48.032917 908 ts_tablet_manager.cc:1367] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20240501 14:01:48.033712 908 raft_consensus.cc:348] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:48.034018 908 raft_consensus.cc:729] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:48.034247 908 consensus_queue.cc:260] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:48.035096 908 ts_tablet_manager.cc:1398] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.002s user 0.002s sys 0.004s
I20240501 14:01:48.035997 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-1ccf3fc5fc7f44a3bc53dad7902c2d3a received from {username='slave'} at 127.22.5.199:57317
I20240501 14:01:48.036145 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-1ccf3fc5fc7f44a3bc53dad7902c2d3a on tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:48.150066 31945 raft_consensus.cc:1203] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.537271 1031 tablet_copy_client.cc:523] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.540614 1031 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:48.556923 1041 tablet_copy_client.cc:523] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.560679 1041 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:48.580773 1031 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.581343 1031 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:48.581652 1031 ts_tablet_manager.cc:1367] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.005s
I20240501 14:01:48.582551 1031 raft_consensus.cc:348] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:48.582840 1031 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:48.583071 1031 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:48.583962 1031 ts_tablet_manager.cc:1398] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:01:48.584932 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-594dcb0a05054f2abaa11e5d4537b724 received from {username='slave'} at 127.22.5.197:39969
I20240501 14:01:48.585098 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-594dcb0a05054f2abaa11e5d4537b724 on tablet 594dcb0a05054f2abaa11e5d4537b724 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:48.586093 32075 raft_consensus.cc:1203] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.603402 1041 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.603798 1041 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:48.604061 1041 ts_tablet_manager.cc:1367] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.043s user 0.037s sys 0.005s
I20240501 14:01:48.604714 1041 raft_consensus.cc:348] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:48.604925 1041 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:48.605168 1041 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:48.605897 1041 ts_tablet_manager.cc:1398] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:48.606742 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-31303ad8c9124217a45e26f8da5cc0a1 received from {username='slave'} at 127.22.5.200:42889
I20240501 14:01:48.606915 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 744cef852ee9491db3928fa0092ef189-31303ad8c9124217a45e26f8da5cc0a1 on tablet 31303ad8c9124217a45e26f8da5cc0a1 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:48.633782 1077 tablet_copy_client.cc:523] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.637602 1077 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:48.672010 31812 raft_consensus.cc:1203] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.673164 1077 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.673560 1077 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:48.673821 1077 ts_tablet_manager.cc:1367] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.012s
I20240501 14:01:48.674572 1077 raft_consensus.cc:348] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:48.674789 1077 raft_consensus.cc:729] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:48.674983 1077 consensus_queue.cc:260] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:48.675443 1077 ts_tablet_manager.cc:1398] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:48.676275 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-9b56df9dcda34e6a8d27a8a6b24742b2 received from {username='slave'} at 127.22.5.200:49589
I20240501 14:01:48.676386 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 744cef852ee9491db3928fa0092ef189-9b56df9dcda34e6a8d27a8a6b24742b2 on tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:48.687887 1106 tablet_copy_client.cc:523] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.691764 1106 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:48.711167 32210 raft_consensus.cc:1203] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.724906 1106 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.725029 716 raft_consensus.cc:1050] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:48.725468 1106 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:48.725795 1106 ts_tablet_manager.cc:1367] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.007s
I20240501 14:01:48.726004 716 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:48.726626 1106 raft_consensus.cc:348] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:48.726914 1106 raft_consensus.cc:729] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:48.727141 1106 consensus_queue.cc:260] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:48.729027 30162 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:48.729077 31945 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:48.730157 1106 ts_tablet_manager.cc:1398] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:01:48.730243 629 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:48.730576 717 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:48.731858 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-f4988f0de3214860a65007b8e31e88a5 received from {username='slave'} at 127.22.5.198:45035
I20240501 14:01:48.732009 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-f4988f0de3214860a65007b8e31e88a5 on tablet f4988f0de3214860a65007b8e31e88a5 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:48.733060 29899 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:48.733795 629 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:48.737085 716 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.737800 31945 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.739362 29899 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.742444 29752 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.742475 30162 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.757491 30032 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:48.759974 31945 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:48.760013 30162 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:48.761476 717 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:48.761845 629 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:48.764801 716 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.765590 30162 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.765529 31945 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.768855 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0929994d486d4610b9cc76abd39f043a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:48.770570 29760 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:48.774727 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0929994d486d4610b9cc76abd39f043a with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:48.775671 1496 tablet_replica.cc:330] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:48.775897 1496 raft_consensus.cc:2229] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:48.776188 1496 raft_consensus.cc:2258] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:48.777295 1496 ts_tablet_manager.cc:1875] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:48.783524 1496 ts_tablet_manager.cc:1888] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:48.783739 1496 log.cc:1196] T 0929994d486d4610b9cc76abd39f043a P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/0929994d486d4610b9cc76abd39f043a
I20240501 14:01:48.784745 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 0929994d486d4610b9cc76abd39f043a (table rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]) successfully deleted
I20240501 14:01:48.808117 1157 tablet_copy_client.cc:523] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.811462 1157 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:48.828648 32210 raft_consensus.cc:1203] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:48.862488 1157 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:48.862880 1157 tablet_bootstrap.cc:492] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:48.863193 1157 ts_tablet_manager.cc:1367] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.052s user 0.043s sys 0.004s
I20240501 14:01:48.863988 1157 raft_consensus.cc:348] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:48.864271 1157 raft_consensus.cc:729] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:48.864503 1157 consensus_queue.cc:260] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:48.865142 1157 ts_tablet_manager.cc:1398] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:48.865883 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-48b5a3abfd8348b9ae47c8867642044e received from {username='slave'} at 127.22.5.197:45235
I20240501 14:01:48.866056 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-48b5a3abfd8348b9ae47c8867642044e on tablet 48b5a3abfd8348b9ae47c8867642044e with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:48.945560 1212 tablet_copy_client.cc:523] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.948781 1212 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:48.964704 1219 tablet_copy_client.cc:523] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.968437 1219 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:48.976058 1224 tablet_copy_client.cc:523] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:48.978816 1224 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:49.014356 1212 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:49.014829 1212 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:49.015156 1212 ts_tablet_manager.cc:1367] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.066s user 0.054s sys 0.007s
I20240501 14:01:49.015965 1212 raft_consensus.cc:348] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:49.016252 1212 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:49.016458 1212 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:49.017916 1212 ts_tablet_manager.cc:1398] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:49.019851 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-c00649352f9d41c3a12a5e767b8b2aa6 received from {username='slave'} at 127.22.5.197:39969
I20240501 14:01:49.020048 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-c00649352f9d41c3a12a5e767b8b2aa6 on tablet c00649352f9d41c3a12a5e767b8b2aa6 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:49.031387 1219 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:49.031921 1219 tablet_bootstrap.cc:492] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:49.032198 1219 ts_tablet_manager.cc:1367] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.064s user 0.038s sys 0.022s
I20240501 14:01:49.033286 1219 raft_consensus.cc:348] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:49.033598 1219 raft_consensus.cc:729] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:49.033821 1219 consensus_queue.cc:260] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:49.034426 1219 ts_tablet_manager.cc:1398] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:49.035878 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-64012de915564c7887e44d432b170fdc received from {username='slave'} at 127.22.5.200:49589
I20240501 14:01:49.036042 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 744cef852ee9491db3928fa0092ef189-64012de915564c7887e44d432b170fdc on tablet 64012de915564c7887e44d432b170fdc with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:49.041795 1224 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:49.042289 1224 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:49.042577 1224 ts_tablet_manager.cc:1367] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.064s user 0.042s sys 0.001s
I20240501 14:01:49.043206 1224 raft_consensus.cc:348] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:49.043413 1224 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:49.043598 1224 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:49.044091 1224 ts_tablet_manager.cc:1398] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:49.044910 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-2b1ada0fe1ce431a823f2814f116db28 received from {username='slave'} at 127.22.5.199:46389
I20240501 14:01:49.045050 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-2b1ada0fe1ce431a823f2814f116db28 on tablet 2b1ada0fe1ce431a823f2814f116db28 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:49.164270 1590 raft_consensus.cc:1050] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:49.165278 1590 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.168228 30032 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.168215 32210 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.168211 30162 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.169075 1590 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.169488 1593 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.169845 1591 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.177543 717 raft_consensus.cc:1050] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:49.178467 717 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.182426 29899 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.182638 31812 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.182993 30291 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.183271 716 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.183576 32513 raft_consensus.cc:1050] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:49.183590 717 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.184376 716 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.184505 32513 consensus_queue.cc:237] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:49.184970 30162 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.183964 685 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.195587 30032 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.206384 29756 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.215456 32209 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.220117 30291 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.223484 30032 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.225636 685 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.232295 685 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.233975 30032 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.247418 29756 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.252830 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:49.257283 32210 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.257861 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.259330 1593 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.261070 32210 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.263346 717 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.264348 30032 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.265066 32513 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.265470 32075 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.265491 29899 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.266578 30291 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.268610 1596 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.269053 31813 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.274405 30291 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.278703 29760 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.280196 1606 tablet_replica.cc:330] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:49.280465 1606 raft_consensus.cc:2229] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.280920 1606 raft_consensus.cc:2258] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.285185 32210 raft_consensus.cc:1203] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:49.285434 32513 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.289341 1606 ts_tablet_manager.cc:1875] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.294387 31946 raft_consensus.cc:1203] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:49.315038 1606 ts_tablet_manager.cc:1888] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:49.315724 1606 log.cc:1196] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:01:49.316926 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 9b56df9dcda34e6a8d27a8a6b24742b2 (table rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]) successfully deleted
I20240501 14:01:49.325181 1596 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.325934 32076 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.329767 30291 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.331456 29756 catalog_manager.cc:5484] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:49.333021 30031 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.341701 31812 raft_consensus.cc:1203] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:49.370234 29899 consensus_queue.cc:237] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:49.372691 30032 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.372879 32076 raft_consensus.cc:1261] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.373267 1616 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.373638 1608 consensus_queue.cc:1035] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.376431 1608 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.377236 30032 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.377466 32076 raft_consensus.cc:2932] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:49.379760 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.382632 29756 catalog_manager.cc:5484] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:49.387084 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:49.387871 1622 tablet_replica.cc:330] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:49.388293 1622 raft_consensus.cc:2229] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.388700 1622 raft_consensus.cc:2258] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.389734 1622 ts_tablet_manager.cc:1875] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.394003 32513 raft_consensus.cc:1050] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:49.394917 32513 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.395218 31812 raft_consensus.cc:1203] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:49.397464 1622 ts_tablet_manager.cc:1888] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:49.397646 1622 log.cc:1196] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ccf3fc5fc7f44a3bc53dad7902c2d3a
I20240501 14:01:49.398561 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a (table rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]) successfully deleted
I20240501 14:01:49.401518 30291 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.401827 30032 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.401912 32210 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.402272 1608 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.402712 32513 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.403046 1616 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.407199 1608 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.407900 30291 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.413100 30032 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.414800 32210 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.416277 29760 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:49.424854 29899 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.426966 32210 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.426961 30031 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.427699 1608 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.428097 1616 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.430578 1608 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.432335 30032 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.433610 32210 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.434685 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31303ad8c9124217a45e26f8da5cc0a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.437963 32076 raft_consensus.cc:1203] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:49.442148 29754 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.447798 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 31303ad8c9124217a45e26f8da5cc0a1 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:49.448175 1622 tablet_replica.cc:330] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:49.448415 1622 raft_consensus.cc:2229] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.448715 1622 raft_consensus.cc:2258] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.449664 1622 ts_tablet_manager.cc:1875] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.456091 1622 ts_tablet_manager.cc:1888] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:49.456218 1622 log.cc:1196] T 31303ad8c9124217a45e26f8da5cc0a1 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:01:49.456970 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 31303ad8c9124217a45e26f8da5cc0a1 (table rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]) successfully deleted
I20240501 14:01:49.466217 30032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39788
I20240501 14:01:49.466516 30032 raft_consensus.cc:595] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:49.563347 30032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39838
I20240501 14:01:49.563565 30032 raft_consensus.cc:595] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:49.685807 716 raft_consensus.cc:979] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: : Instructing follower 7869901e77234aeeb1462876b63eb28e to start an election
I20240501 14:01:49.686034 716 raft_consensus.cc:1067] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Signalling peer 7869901e77234aeeb1462876b63eb28e to start an election
I20240501 14:01:49.687022 29899 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
from {username='slave'} at 127.22.5.194:42221
I20240501 14:01:49.687299 29899 raft_consensus.cc:482] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:49.687433 29899 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:49.690449 29899 raft_consensus.cc:504] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.691411 29899 leader_election.cc:288] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 2 election: Requested vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
I20240501 14:01:49.692145 31812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
I20240501 14:01:49.692281 30291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:49.692477 30291 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:49.692435 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594dcb0a05054f2abaa11e5d4537b724" candidate_uuid: "7869901e77234aeeb1462876b63eb28e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:49.692523 31812 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:49.692631 30032 raft_consensus.cc:3032] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:49.692757 30032 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:49.693106 30032 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.693622 30032 raft_consensus.cc:3037] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:49.695482 30291 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 2.
I20240501 14:01:49.695475 31812 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 2.
I20240501 14:01:49.696699 29836 leader_election.cc:302] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e, be55dad80e3a4ad79d21aaa3199b1d0b; no voters:
I20240501 14:01:49.697110 1616 raft_consensus.cc:2781] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:49.697299 1616 raft_consensus.cc:686] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 LEADER]: Becoming Leader. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:49.697611 1616 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.698204 30032 raft_consensus.cc:2454] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7869901e77234aeeb1462876b63eb28e in term 2.
I20240501 14:01:49.700870 29760 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e reported cstate change: term changed from 1 to 2, leader changed from 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) to 7869901e77234aeeb1462876b63eb28e (127.22.5.193). New cstate: current_term: 2 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:49.802213 1718 raft_consensus.cc:1050] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:49.803036 1718 consensus_queue.cc:237] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.806085 31812 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.806159 30291 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.806252 29899 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.806805 1718 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.807219 32730 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.807698 32730 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.819311 1717 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.824484 30291 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.825794 29760 catalog_manager.cc:5484] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:49.826274 31812 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.824295 29901 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.836757 30162 consensus_queue.cc:237] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.839247 31812 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.840513 1717 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.841761 29901 raft_consensus.cc:1261] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.842418 1717 consensus_queue.cc:1035] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.844231 1717 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.844792 31812 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.847131 29901 raft_consensus.cc:2932] T 48b5a3abfd8348b9ae47c8867642044e P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.849316 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48b5a3abfd8348b9ae47c8867642044e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.851207 29759 catalog_manager.cc:5484] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 5 to 6, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:49.855551 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b5a3abfd8348b9ae47c8867642044e with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:49.856200 1622 tablet_replica.cc:330] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:49.856427 1622 raft_consensus.cc:2229] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.856802 1622 raft_consensus.cc:2258] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.857774 1622 ts_tablet_manager.cc:1875] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.862898 1622 ts_tablet_manager.cc:1888] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:49.863104 1622 log.cc:1196] T 48b5a3abfd8348b9ae47c8867642044e P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/48b5a3abfd8348b9ae47c8867642044e
I20240501 14:01:49.863832 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 48b5a3abfd8348b9ae47c8867642044e (table rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]) successfully deleted
I20240501 14:01:49.871354 717 raft_consensus.cc:1050] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:49.872284 717 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:49.876176 31946 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.876863 30162 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.877503 1734 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.878135 30291 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.879876 1736 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.882349 1734 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.885226 716 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.885922 31946 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.889394 1593 raft_consensus.cc:1050] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:49.890604 1593 consensus_queue.cc:237] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.891409 30162 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.903405 29754 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:49.907169 30291 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.910858 30162 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.911022 32210 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.911772 1593 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.912068 30032 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.912186 1591 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.913100 1591 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.916769 685 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.917845 30032 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.920785 32210 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.921556 30031 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:49.923287 31946 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.924918 1734 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.925916 29754 catalog_manager.cc:5484] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.930289 30290 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.931478 716 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.931932 716 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.932802 31946 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.936213 30290 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.936846 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4988f0de3214860a65007b8e31e88a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.938498 29754 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:49.942412 30162 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.943581 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet f4988f0de3214860a65007b8e31e88a5 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:49.949554 1752 tablet_replica.cc:330] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:49.949779 1752 raft_consensus.cc:2229] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:49.950150 1752 raft_consensus.cc:2258] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:49.950999 1752 ts_tablet_manager.cc:1875] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:49.954685 30290 consensus_queue.cc:237] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:49.956291 1736 raft_consensus.cc:1050] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:49.956998 1736 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:49.957418 1752 ts_tablet_manager.cc:1888] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:49.957540 1752 log.cc:1196] T f4988f0de3214860a65007b8e31e88a5 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4988f0de3214860a65007b8e31e88a5
I20240501 14:01:49.958002 32210 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.958485 30162 raft_consensus.cc:1261] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:49.958612 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet f4988f0de3214860a65007b8e31e88a5 (table rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]) successfully deleted
I20240501 14:01:49.959120 1750 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.960458 1593 consensus_queue.cc:1035] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:49.961992 1591 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.962396 31812 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.962656 32210 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.963025 717 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.963197 30161 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.965940 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64012de915564c7887e44d432b170fdc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:49.966702 717 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.969446 29754 catalog_manager.cc:5484] T 64012de915564c7887e44d432b170fdc P 744cef852ee9491db3928fa0092ef189 reported cstate change: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.969280 30162 raft_consensus.cc:2932] T 64012de915564c7887e44d432b170fdc P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:49.975369 29901 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.976600 716 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.978750 717 raft_consensus.cc:1050] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:49.979722 717 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:49.982712 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 64012de915564c7887e44d432b170fdc with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:49.983839 716 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.986397 31812 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.988286 32076 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.988080 30161 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.989109 1734 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.989106 29901 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.993023 29756 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 4 to 5, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:49.996850 29901 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:49.997411 1759 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:49.998637 30290 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:50.000530 30031 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:50.001848 1736 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:50.002494 29899 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:50.003126 1736 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.004282 1758 tablet_replica.cc:330] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:50.004406 31812 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:50.004657 1758 raft_consensus.cc:2229] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:50.005100 1758 raft_consensus.cc:2258] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:50.005069 1736 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.006369 1758 ts_tablet_manager.cc:1875] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:50.007584 716 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.007074 1759 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.008633 29899 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.012567 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c00649352f9d41c3a12a5e767b8b2aa6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:50.012615 32076 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.012655 31812 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.013216 29901 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.012440 30290 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.018258 29759 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.022490 29760 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 5 to 6, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:50.027091 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet c00649352f9d41c3a12a5e767b8b2aa6 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:50.027500 1752 tablet_replica.cc:330] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:50.027722 1752 raft_consensus.cc:2229] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:50.028059 1752 raft_consensus.cc:2258] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:50.028688 1758 ts_tablet_manager.cc:1888] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:50.028894 1758 log.cc:1196] T 64012de915564c7887e44d432b170fdc P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/64012de915564c7887e44d432b170fdc
I20240501 14:01:50.028995 1752 ts_tablet_manager.cc:1875] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:50.029901 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 64012de915564c7887e44d432b170fdc (table rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]) successfully deleted
I20240501 14:01:50.034189 1752 ts_tablet_manager.cc:1888] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:01:50.034307 1752 log.cc:1196] T c00649352f9d41c3a12a5e767b8b2aa6 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:01:50.034972 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet c00649352f9d41c3a12a5e767b8b2aa6 (table rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]) successfully deleted
I20240501 14:01:50.093211 31812 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.093870 1608 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.199846 30031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39930
I20240501 14:01:50.200058 30031 raft_consensus.cc:595] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:50.234057 30031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39954
I20240501 14:01:50.234278 30031 raft_consensus.cc:595] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:50.271173 30290 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.271731 32513 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.276026 30031 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.276659 32513 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.279666 29899 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:50.287012 31812 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:50.288367 1815 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:50.290159 30291 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:50.291337 32513 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:50.292176 32513 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.294776 30290 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.296537 31812 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:50.297403 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 594dcb0a05054f2abaa11e5d4537b724 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:50.300985 29754 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 2 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:50.305310 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 594dcb0a05054f2abaa11e5d4537b724 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:50.305685 1758 tablet_replica.cc:330] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:50.305920 1758 raft_consensus.cc:2229] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:50.306108 1758 raft_consensus.cc:2258] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:50.306933 1758 ts_tablet_manager.cc:1875] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:50.318317 1758 ts_tablet_manager.cc:1888] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:01:50.318485 1758 log.cc:1196] T 594dcb0a05054f2abaa11e5d4537b724 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/594dcb0a05054f2abaa11e5d4537b724
I20240501 14:01:50.319202 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 594dcb0a05054f2abaa11e5d4537b724 (table rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]) successfully deleted
I20240501 14:01:50.464313 30031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39964
I20240501 14:01:50.464601 30031 raft_consensus.cc:595] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:50.483222 30031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39972
I20240501 14:01:50.483450 30031 raft_consensus.cc:595] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:50.487701 716 raft_consensus.cc:979] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: : Instructing follower 47b7351d2e9b4dbd9a09fd4133a97565 to start an election
I20240501 14:01:50.487871 716 raft_consensus.cc:1067] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Signalling peer 47b7351d2e9b4dbd9a09fd4133a97565 to start an election
I20240501 14:01:50.488500 32076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28"
dest_uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
from {username='slave'} at 127.22.5.194:38779
I20240501 14:01:50.488740 32076 raft_consensus.cc:482] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:50.488898 32076 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:50.490936 32076 raft_consensus.cc:504] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:50.492642 32076 leader_election.cc:288] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 election: Requested vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:50.493016 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:50.493063 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a15e881772b41eab8a71fc027390db9"
I20240501 14:01:50.493265 30031 raft_consensus.cc:3032] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:50.493314 30290 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:50.493352 30031 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:50.493314 29899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1ada0fe1ce431a823f2814f116db28" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:50.493525 29899 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:50.493643 30031 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:50.494057 30031 raft_consensus.cc:3037] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:50.495153 30290 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:50.496280 29899 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:50.496743 32010 leader_election.cc:302] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47b7351d2e9b4dbd9a09fd4133a97565, 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:50.496812 30031 raft_consensus.cc:2454] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:50.497195 1763 raft_consensus.cc:2781] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:50.497826 1763 raft_consensus.cc:686] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Becoming Leader. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Running, Role: LEADER
I20240501 14:01:50.498226 1763 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:50.502300 29760 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: term changed from 1 to 2, leader changed from 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) to 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199). New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:50.887223 29899 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.887907 1763 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.968209 30291 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.968822 1763 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.973204 30031 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:50.973872 1859 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:50.984261 32076 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:50.986269 29899 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:50.986384 30291 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:50.986810 1859 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:50.987035 1763 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:50.989440 1763 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.990010 30291 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.990000 29899 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:50.994889 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b1ada0fe1ce431a823f2814f116db28 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:50.997936 29760 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 5 to 7, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:51.001672 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1ada0fe1ce431a823f2814f116db28 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:51.002831 1865 tablet_replica.cc:330] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:51.003049 1865 raft_consensus.cc:2229] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:51.003314 1865 raft_consensus.cc:2258] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:51.004105 1865 ts_tablet_manager.cc:1875] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:51.008802 1865 ts_tablet_manager.cc:1888] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:01:51.009071 1865 log.cc:1196] T 2b1ada0fe1ce431a823f2814f116db28 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b1ada0fe1ce431a823f2814f116db28
I20240501 14:01:51.010136 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 2b1ada0fe1ce431a823f2814f116db28 (table rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]) successfully deleted
I20240501 14:01:51.247478 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.273783 32057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:51.278898 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.287575 30270 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:51.311071 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:51.322633 29880 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:51.362617 30010 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:01:51.411427 547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.429344 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.503861 29899 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:51.506969 30031 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:51.507501 30291 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:51.508066 1608 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:51.508474 32513 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:51.528144 1608 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
W20240501 14:01:51.531822 29835 consensus_peers.cc:485] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2b6d21ea3184a6eb9db2b87589781b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.533937 30291 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.536805 30031 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.539109 29756 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index -1 to 4, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.564685 30031 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:51.567613 30291 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.567726 32076 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.568176 29968 consensus_peers.cc:485] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 427af2a000f34ab6b9e3c9c7d7be1b32. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.568321 1864 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.569016 1930 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.571636 1864 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.572355 30291 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.573727 32076 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.578539 29757 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.593644 30291 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
W20240501 14:01:51.596153 30225 consensus_peers.cc:485] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b56df9dcda34e6a8d27a8a6b24742b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.596658 32210 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.597239 30031 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.597476 1593 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.598486 1593 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.601287 1926 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.604760 32210 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.604940 30031 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.610749 29753 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.635898 30031 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:51.638499 31946 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.639248 30161 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.639180 1864 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.639878 1864 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.643299 1937 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.645493 30161 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.650755 29756 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.672544 31946 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
W20240501 14:01:51.673167 29966 consensus_peers.cc:485] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0929994d486d4610b9cc76abd39f043a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.687769 29899 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:51.690145 30031 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.690153 32210 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.690335 29836 consensus_peers.cc:485] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31303ad8c9124217a45e26f8da5cc0a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.690819 1608 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.691152 32513 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.694141 1608 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.698139 30031 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.699460 29757 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.711908 32210 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.728170 30031 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:51.730846 31946 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.730846 30291 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.731245 29968 consensus_peers.cc:485] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4988f0de3214860a65007b8e31e88a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.731593 1864 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.731925 1937 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.734642 1864 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.735357 30291 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.735872 31946 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.741101 29756 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 6 to 7, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.754182 30291 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:51.756747 32076 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.756872 30031 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.757514 1949 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.757867 1593 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.761404 1593 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.763221 30031 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
W20240501 14:01:51.764571 30228 consensus_peers.cc:485] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c40a79b118424a5699bddd1c4ac4bb0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.764971 32076 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.770552 29757 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.784305 30161 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:51.786885 31946 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.787289 30291 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.787384 30095 consensus_peers.cc:485] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98d94ed11f0f4a9d9f519a24c3320ce4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.787817 1993 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.788291 1720 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.791378 1994 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.792156 30291 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.792217 31946 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.797814 29752 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 6 to 7, NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } } }
I20240501 14:01:51.818137 29899 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:51.821095 31946 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.821182 30161 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.821571 29835 consensus_peers.cc:485] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 368edc181c894686a213fc542f328192. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.821897 1608 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.822201 32513 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.824719 1608 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.827744 30161 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.830113 29756 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.839478 31946 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.852283 30291 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
W20240501 14:01:51.855052 30227 consensus_peers.cc:485] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56f632889b1e400392fe710bf9f6f3ec. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.855096 29899 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.856019 31812 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.856201 1946 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.857232 1946 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.860419 1949 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.860873 29899 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.864552 31813 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:01:51.866050 29757 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 6 to 7, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:51.886991 32076 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:51.889647 29899 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:01:51.890439 1763 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:51.894335 1763 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.899534 2023 ts_tablet_manager.cc:902] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:51.899808 29899 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.909538 2023 tablet_copy_client.cc:308] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:51.910503 29752 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.921871 30291 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:01:51.922595 2028 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:51.924791 30291 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 744cef852ee9491db3928fa0092ef189 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } } }
I20240501 14:01:51.931895 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet f2b6d21ea3184a6eb9db2b87589781b0 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:51.932124 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet f2b6d21ea3184a6eb9db2b87589781b0 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:01:51.936223 29921 tablet_copy_source_session.cc:205] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:01:51.941526 32013 consensus_peers.cc:485] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 -> Peer 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981): Couldn't send request to peer 744cef852ee9491db3928fa0092ef189. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1ada0fe1ce431a823f2814f116db28. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.942188 2023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d21ea3184a6eb9db2b87589781b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:51.949404 29899 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:51.952087 31946 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.952075 29835 consensus_peers.cc:485] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ea9aaebd437424eb27c04bc2f6ffa7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.952087 30031 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.952842 1608 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.953197 32513 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.955444 2023 tablet_copy_client.cc:783] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:51.955832 2023 tablet_copy_client.cc:647] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:51.956458 1608 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.957257 30031 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.958117 2027 tablet_copy_client.cc:1002] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:51.962951 29752 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.978826 31946 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:51.988620 29899 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:51.992039 32210 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:51.992131 30161 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:51.992463 29837 consensus_peers.cc:485] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c35ccca22fb45b78ee66991027c0261. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:51.992763 2040 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.993486 2039 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:51.995975 1608 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.999531 32210 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:51.999647 30161 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.005553 29756 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 6 to 7, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.018848 29899 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:52.021242 31813 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:01:52.021415 29837 consensus_peers.cc:485] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e -> Peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Couldn't send request to peer 3321c47e58ba49f09f4f036ac72fc133. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 594dcb0a05054f2abaa11e5d4537b724. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:52.021454 30291 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:01:52.021847 1922 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:52.022395 1922 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:52.024621 32513 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.025290 31813 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.025904 30291 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.029950 29757 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 7 to 8, NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) added. New cstate: current_term: 2 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } } }
I20240501 14:01:52.032976 2057 ts_tablet_manager.cc:902] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:52.046507 2057 tablet_copy_client.cc:308] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:52.047370 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:45035)
I20240501 14:01:52.047569 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 427af2a000f34ab6b9e3c9c7d7be1b32 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:45035: session id = 3321c47e58ba49f09f4f036ac72fc133-427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:01:52.050230 30052 tablet_copy_source_session.cc:205] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.052196 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427af2a000f34ab6b9e3c9c7d7be1b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.055619 30031 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:52.057865 2057 tablet_copy_client.cc:783] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.058140 2057 tablet_copy_client.cc:647] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
W20240501 14:01:52.058359 29966 consensus_peers.cc:485] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c00649352f9d41c3a12a5e767b8b2aa6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:52.058385 29899 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:52.058965 31813 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:52.058941 1930 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:52.059615 1930 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:52.059998 2066 tablet_copy_client.cc:1002] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.062011 1937 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:52.062634 29899 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:52.063009 31813 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:52.067157 29752 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:52.082197 2071 ts_tablet_manager.cc:902] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:52.083863 2071 tablet_copy_client.cc:308] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:52.084765 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:37661)
I20240501 14:01:52.084937 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet 9b56df9dcda34e6a8d27a8a6b24742b2 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:37661: session id = be55dad80e3a4ad79d21aaa3199b1d0b-9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:01:52.087109 30312 tablet_copy_source_session.cc:205] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.088398 2071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56df9dcda34e6a8d27a8a6b24742b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.094451 2071 tablet_copy_client.cc:783] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.094837 2071 tablet_copy_client.cc:647] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.096841 2076 tablet_copy_client.cc:1002] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.125432 2095 ts_tablet_manager.cc:902] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:52.127403 2095 tablet_copy_client.cc:308] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:52.128152 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 31303ad8c9124217a45e26f8da5cc0a1 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:57317)
I20240501 14:01:52.128297 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 31303ad8c9124217a45e26f8da5cc0a1 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:57317: session id = 47b7351d2e9b4dbd9a09fd4133a97565-31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:01:52.130923 29921 tablet_copy_source_session.cc:205] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.132288 2095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31303ad8c9124217a45e26f8da5cc0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.138422 2105 ts_tablet_manager.cc:902] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:52.139173 2095 tablet_copy_client.cc:783] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.139451 2095 tablet_copy_client.cc:647] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.141320 2099 tablet_copy_client.cc:1002] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.142221 2105 tablet_copy_client.cc:308] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:52.148710 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet 0929994d486d4610b9cc76abd39f043a from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:50499)
I20240501 14:01:52.148893 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet 0929994d486d4610b9cc76abd39f043a from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:50499: session id = aa11715897034babb1009134d8914066-0929994d486d4610b9cc76abd39f043a
I20240501 14:01:52.151491 30052 tablet_copy_source_session.cc:205] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.154376 2105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929994d486d4610b9cc76abd39f043a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.159575 2105 tablet_copy_client.cc:783] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.159857 2105 tablet_copy_client.cc:647] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.162678 2109 tablet_copy_client.cc:1002] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.197156 2135 ts_tablet_manager.cc:902] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:52.218824 2135 tablet_copy_client.cc:308] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:52.233553 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet c40a79b118424a5699bddd1c4ac4bb0e from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:55855)
I20240501 14:01:52.233765 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet c40a79b118424a5699bddd1c4ac4bb0e from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:55855: session id = 3321c47e58ba49f09f4f036ac72fc133-c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:01:52.235929 30312 tablet_copy_source_session.cc:205] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.239816 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a79b118424a5699bddd1c4ac4bb0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.245846 2135 tablet_copy_client.cc:783] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.246217 2135 tablet_copy_client.cc:647] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.248517 2145 tablet_copy_client.cc:1002] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.305392 2167 ts_tablet_manager.cc:902] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:52.307675 2167 tablet_copy_client.cc:308] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:52.308586 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet f4988f0de3214860a65007b8e31e88a5 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:36943)
I20240501 14:01:52.308804 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet f4988f0de3214860a65007b8e31e88a5 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:36943: session id = 744cef852ee9491db3928fa0092ef189-f4988f0de3214860a65007b8e31e88a5
I20240501 14:01:52.311749 30052 tablet_copy_source_session.cc:205] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.313925 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4988f0de3214860a65007b8e31e88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.319320 2167 tablet_copy_client.cc:783] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.319559 2167 tablet_copy_client.cc:647] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.321178 2171 tablet_copy_client.cc:1002] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.347481 2172 ts_tablet_manager.cc:902] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:52.349279 2172 tablet_copy_client.cc:308] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:52.350167 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet 56f632889b1e400392fe710bf9f6f3ec from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:33017)
I20240501 14:01:52.350327 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet 56f632889b1e400392fe710bf9f6f3ec from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:33017: session id = 47b7351d2e9b4dbd9a09fd4133a97565-56f632889b1e400392fe710bf9f6f3ec
I20240501 14:01:52.352905 30312 tablet_copy_source_session.cc:205] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.353912 2177 ts_tablet_manager.cc:902] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Initiating tablet copy from peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447)
I20240501 14:01:52.354492 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f632889b1e400392fe710bf9f6f3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.361795 2172 tablet_copy_client.cc:783] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.362097 2172 tablet_copy_client.cc:647] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.364826 2178 ts_tablet_manager.cc:902] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:52.365592 2176 tablet_copy_client.cc:1002] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.366724 2177 tablet_copy_client.cc:308] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:40447
I20240501 14:01:52.375635 32097 tablet_copy_service.cc:140] P 47b7351d2e9b4dbd9a09fd4133a97565: Received BeginTabletCopySession request for tablet 2b1ada0fe1ce431a823f2814f116db28 from peer 744cef852ee9491db3928fa0092ef189 ({username='slave'} at 127.22.5.200:56695)
I20240501 14:01:52.375880 32097 tablet_copy_service.cc:161] P 47b7351d2e9b4dbd9a09fd4133a97565: Beginning new tablet copy session on tablet 2b1ada0fe1ce431a823f2814f116db28 from peer 744cef852ee9491db3928fa0092ef189 at {username='slave'} at 127.22.5.200:56695: session id = 744cef852ee9491db3928fa0092ef189-2b1ada0fe1ce431a823f2814f116db28
I20240501 14:01:52.378443 2178 tablet_copy_client.cc:308] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:52.379046 32097 tablet_copy_source_session.cc:205] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.379225 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 368edc181c894686a213fc542f328192 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:52.379420 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 368edc181c894686a213fc542f328192 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-368edc181c894686a213fc542f328192
I20240501 14:01:52.380872 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1ada0fe1ce431a823f2814f116db28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.381774 29921 tablet_copy_source_session.cc:205] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.383062 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368edc181c894686a213fc542f328192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.387029 2177 tablet_copy_client.cc:783] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.387288 2177 tablet_copy_client.cc:647] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.389103 2178 tablet_copy_client.cc:783] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.389305 2182 tablet_copy_client.cc:1002] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.389400 2178 tablet_copy_client.cc:647] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.390982 2188 tablet_copy_client.cc:1002] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.403016 2189 ts_tablet_manager.cc:902] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:52.404932 2189 tablet_copy_client.cc:308] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:52.405706 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 from peer be55dad80e3a4ad79d21aaa3199b1d0b ({username='slave'} at 127.22.5.197:45235)
I20240501 14:01:52.405865 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 98d94ed11f0f4a9d9f519a24c3320ce4 from peer be55dad80e3a4ad79d21aaa3199b1d0b at {username='slave'} at 127.22.5.197:45235: session id = be55dad80e3a4ad79d21aaa3199b1d0b-98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:01:52.408217 30182 tablet_copy_source_session.cc:205] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.409652 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d94ed11f0f4a9d9f519a24c3320ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.414168 2189 tablet_copy_client.cc:783] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.414427 2189 tablet_copy_client.cc:647] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.416329 2193 tablet_copy_client.cc:1002] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.457084 2194 ts_tablet_manager.cc:902] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:52.458784 2194 tablet_copy_client.cc:308] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:52.459540 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 5c35ccca22fb45b78ee66991027c0261 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:37341)
I20240501 14:01:52.459781 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 5c35ccca22fb45b78ee66991027c0261 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:37341: session id = 3321c47e58ba49f09f4f036ac72fc133-5c35ccca22fb45b78ee66991027c0261
I20240501 14:01:52.462035 29921 tablet_copy_source_session.cc:205] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.463454 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c35ccca22fb45b78ee66991027c0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.467958 2194 tablet_copy_client.cc:783] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.468200 2194 tablet_copy_client.cc:647] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.469871 2198 tablet_copy_client.cc:1002] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.494354 2205 ts_tablet_manager.cc:902] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:52.497555 2205 tablet_copy_client.cc:308] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:52.498361 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 594dcb0a05054f2abaa11e5d4537b724 from peer 3321c47e58ba49f09f4f036ac72fc133 ({username='slave'} at 127.22.5.198:37341)
I20240501 14:01:52.498533 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 594dcb0a05054f2abaa11e5d4537b724 from peer 3321c47e58ba49f09f4f036ac72fc133 at {username='slave'} at 127.22.5.198:37341: session id = 3321c47e58ba49f09f4f036ac72fc133-594dcb0a05054f2abaa11e5d4537b724
I20240501 14:01:52.500659 29921 tablet_copy_source_session.cc:205] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.502673 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594dcb0a05054f2abaa11e5d4537b724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.509085 2205 tablet_copy_client.cc:783] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.509441 2205 tablet_copy_client.cc:647] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.510052 2219 ts_tablet_manager.cc:902] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:52.511124 2213 tablet_copy_client.cc:1002] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.519961 2219 tablet_copy_client.cc:308] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:52.520681 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:52.520849 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 0ea9aaebd437424eb27c04bc2f6ffa7b from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:01:52.523478 29921 tablet_copy_source_session.cc:205] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.524747 2219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.529744 2219 tablet_copy_client.cc:783] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.530158 2219 tablet_copy_client.cc:647] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.532541 2223 tablet_copy_client.cc:1002] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:52.649694 2266 ts_tablet_manager.cc:902] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
I20240501 14:01:52.675119 2266 tablet_copy_client.cc:308] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:38885
I20240501 14:01:52.676026 30052 tablet_copy_service.cc:140] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Received BeginTabletCopySession request for tablet c00649352f9d41c3a12a5e767b8b2aa6 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:50499)
I20240501 14:01:52.676226 30052 tablet_copy_service.cc:161] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Beginning new tablet copy session on tablet c00649352f9d41c3a12a5e767b8b2aa6 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:50499: session id = aa11715897034babb1009134d8914066-c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:01:52.679371 30052 tablet_copy_source_session.cc:205] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:52.680641 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00649352f9d41c3a12a5e767b8b2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:52.686494 2266 tablet_copy_client.cc:783] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:52.686774 2266 tablet_copy_client.cc:647] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:52.689510 2273 tablet_copy_client.cc:1002] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:53.458988 2023 tablet_copy_client.cc:523] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.463486 2023 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:53.473613 2023 log.cc:824] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:53.506989 2023 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.507470 2023 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:53.507900 2023 ts_tablet_manager.cc:1367] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.009s
I20240501 14:01:53.512547 2023 raft_consensus.cc:348] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.513013 2023 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:53.513545 2023 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.514946 2023 ts_tablet_manager.cc:1398] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240501 14:01:53.515939 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-f2b6d21ea3184a6eb9db2b87589781b0 received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:53.516090 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-f2b6d21ea3184a6eb9db2b87589781b0 on tablet f2b6d21ea3184a6eb9db2b87589781b0 with peer aa11715897034babb1009134d8914066
I20240501 14:01:53.560388 2057 tablet_copy_client.cc:523] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.562983 2057 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:53.593384 2057 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.593910 2057 tablet_bootstrap.cc:492] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:53.594213 2057 ts_tablet_manager.cc:1367] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.005s
I20240501 14:01:53.594863 2057 raft_consensus.cc:348] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:53.595099 2057 raft_consensus.cc:729] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:53.595295 2057 consensus_queue.cc:260] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:53.595984 2057 ts_tablet_manager.cc:1398] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:01:53.596856 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-427af2a000f34ab6b9e3c9c7d7be1b32 received from {username='slave'} at 127.22.5.198:45035
I20240501 14:01:53.597044 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-427af2a000f34ab6b9e3c9c7d7be1b32 on tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:53.597249 2071 tablet_copy_client.cc:523] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.600551 2071 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:53.636409 2071 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.636822 2071 tablet_bootstrap.cc:492] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:53.637079 2071 ts_tablet_manager.cc:1367] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.037s user 0.033s sys 0.000s
I20240501 14:01:53.637775 2071 raft_consensus.cc:348] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:53.637989 2071 raft_consensus.cc:729] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:53.638236 2071 consensus_queue.cc:260] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:53.639029 2071 ts_tablet_manager.cc:1398] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:53.639747 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-9b56df9dcda34e6a8d27a8a6b24742b2 received from {username='slave'} at 127.22.5.197:37661
I20240501 14:01:53.639894 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-9b56df9dcda34e6a8d27a8a6b24742b2 on tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:53.641911 2095 tablet_copy_client.cc:523] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.645342 2095 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:53.663062 2105 tablet_copy_client.cc:523] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.666632 2105 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:53.678831 575 raft_consensus.cc:1203] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:01:53.714991 2095 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.715479 2095 tablet_bootstrap.cc:492] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:53.715806 2095 ts_tablet_manager.cc:1367] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.071s user 0.047s sys 0.005s
I20240501 14:01:53.716665 2095 raft_consensus.cc:348] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:53.716956 2095 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:53.717203 2095 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:53.717811 2095 ts_tablet_manager.cc:1398] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:53.718618 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-31303ad8c9124217a45e26f8da5cc0a1 received from {username='slave'} at 127.22.5.199:57317
I20240501 14:01:53.718767 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-31303ad8c9124217a45e26f8da5cc0a1 on tablet 31303ad8c9124217a45e26f8da5cc0a1 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:53.724069 2105 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.724550 2105 tablet_bootstrap.cc:492] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:53.724886 2105 ts_tablet_manager.cc:1367] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.058s user 0.040s sys 0.003s
I20240501 14:01:53.726025 2105 raft_consensus.cc:348] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.726256 2105 raft_consensus.cc:729] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:53.726445 2105 consensus_queue.cc:260] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.727041 2105 ts_tablet_manager.cc:1398] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:53.728788 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session aa11715897034babb1009134d8914066-0929994d486d4610b9cc76abd39f043a received from {username='slave'} at 127.22.5.201:50499
I20240501 14:01:53.728950 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session aa11715897034babb1009134d8914066-0929994d486d4610b9cc76abd39f043a on tablet 0929994d486d4610b9cc76abd39f043a with peer aa11715897034babb1009134d8914066
I20240501 14:01:53.748968 2135 tablet_copy_client.cc:523] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.752192 2135 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:53.796336 2135 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.796774 2135 tablet_bootstrap.cc:492] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:53.797070 2135 ts_tablet_manager.cc:1367] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.045s user 0.031s sys 0.012s
I20240501 14:01:53.797945 2135 raft_consensus.cc:348] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:53.798178 2135 raft_consensus.cc:729] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:53.798375 2135 consensus_queue.cc:260] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:53.798979 2135 ts_tablet_manager.cc:1398] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:53.800460 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-c40a79b118424a5699bddd1c4ac4bb0e received from {username='slave'} at 127.22.5.198:55855
I20240501 14:01:53.800626 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-c40a79b118424a5699bddd1c4ac4bb0e on tablet c40a79b118424a5699bddd1c4ac4bb0e with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:53.815481 31812 raft_consensus.cc:1203] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:53.821655 2167 tablet_copy_client.cc:523] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.825217 2167 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:53.866040 2172 tablet_copy_client.cc:523] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.870466 2172 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:01:53.878038 2167 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.878520 2167 tablet_bootstrap.cc:492] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:53.878834 2167 ts_tablet_manager.cc:1367] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.054s user 0.049s sys 0.004s
I20240501 14:01:53.879648 2167 raft_consensus.cc:348] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:53.879940 2167 raft_consensus.cc:729] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:53.880149 2167 consensus_queue.cc:260] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:53.889693 2177 tablet_copy_client.cc:523] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.892333 2178 tablet_copy_client.cc:523] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.895325 2178 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:53.903110 2167 ts_tablet_manager.cc:1398] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.024s user 0.000s sys 0.014s
I20240501 14:01:53.904103 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-f4988f0de3214860a65007b8e31e88a5 received from {username='slave'} at 127.22.5.200:36943
I20240501 14:01:53.904282 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session 744cef852ee9491db3928fa0092ef189-f4988f0de3214860a65007b8e31e88a5 on tablet f4988f0de3214860a65007b8e31e88a5 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:53.904330 2177 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Bootstrap starting.
I20240501 14:01:53.916751 2189 tablet_copy_client.cc:523] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.922550 2189 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap starting.
I20240501 14:01:53.950402 2172 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.952786 2172 tablet_bootstrap.cc:492] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:01:53.953524 2172 ts_tablet_manager.cc:1367] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.083s user 0.052s sys 0.006s
I20240501 14:01:53.954818 2172 raft_consensus.cc:348] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:53.955591 2172 raft_consensus.cc:729] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:01:53.956174 2172 consensus_queue.cc:260] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:01:53.957188 2172 ts_tablet_manager.cc:1398] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:53.958264 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-56f632889b1e400392fe710bf9f6f3ec received from {username='slave'} at 127.22.5.199:33017
I20240501 14:01:53.958396 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-56f632889b1e400392fe710bf9f6f3ec on tablet 56f632889b1e400392fe710bf9f6f3ec with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:01:53.964987 2178 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.965515 2178 tablet_bootstrap.cc:492] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:53.965816 2178 ts_tablet_manager.cc:1367] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.071s user 0.053s sys 0.004s
I20240501 14:01:53.966706 2178 raft_consensus.cc:348] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.967036 2178 raft_consensus.cc:729] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:53.967267 2178 consensus_queue.cc:260] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:53.967916 2178 ts_tablet_manager.cc:1398] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:53.968688 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-368edc181c894686a213fc542f328192 received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:53.968832 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-368edc181c894686a213fc542f328192 on tablet 368edc181c894686a213fc542f328192 with peer aa11715897034babb1009134d8914066
I20240501 14:01:53.970541 2194 tablet_copy_client.cc:523] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:53.975116 2194 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:53.985549 2177 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:53.986030 2177 tablet_bootstrap.cc:492] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Bootstrap complete.
I20240501 14:01:53.986322 2177 ts_tablet_manager.cc:1367] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Time spent bootstrapping tablet: real 0.082s user 0.052s sys 0.008s
I20240501 14:01:53.987185 2177 raft_consensus.cc:348] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:53.987475 2177 raft_consensus.cc:729] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 744cef852ee9491db3928fa0092ef189, State: Initialized, Role: LEARNER
I20240501 14:01:53.987701 2177 consensus_queue.cc:260] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: true } }
I20240501 14:01:53.988456 2177 ts_tablet_manager.cc:1398] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:53.989591 32097 tablet_copy_service.cc:342] P 47b7351d2e9b4dbd9a09fd4133a97565: Request end of tablet copy session 744cef852ee9491db3928fa0092ef189-2b1ada0fe1ce431a823f2814f116db28 received from {username='slave'} at 127.22.5.200:56695
I20240501 14:01:53.989786 32097 tablet_copy_service.cc:433] P 47b7351d2e9b4dbd9a09fd4133a97565: ending tablet copy session 744cef852ee9491db3928fa0092ef189-2b1ada0fe1ce431a823f2814f116db28 on tablet 2b1ada0fe1ce431a823f2814f116db28 with peer 744cef852ee9491db3928fa0092ef189
I20240501 14:01:53.999636 2189 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.000046 2189 tablet_bootstrap.cc:492] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Bootstrap complete.
I20240501 14:01:54.000376 2189 ts_tablet_manager.cc:1367] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent bootstrapping tablet: real 0.078s user 0.045s sys 0.006s
I20240501 14:01:54.001085 2189 raft_consensus.cc:348] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:54.001380 2189 raft_consensus.cc:729] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55dad80e3a4ad79d21aaa3199b1d0b, State: Initialized, Role: LEARNER
I20240501 14:01:54.001600 2189 consensus_queue.cc:260] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: true } }
I20240501 14:01:54.002188 2189 ts_tablet_manager.cc:1398] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 14:01:54.003151 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-98d94ed11f0f4a9d9f519a24c3320ce4 received from {username='slave'} at 127.22.5.197:45235
I20240501 14:01:54.003284 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session be55dad80e3a4ad79d21aaa3199b1d0b-98d94ed11f0f4a9d9f519a24c3320ce4 on tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with peer be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:01:54.011502 2205 tablet_copy_client.cc:523] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.014312 2205 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap starting.
I20240501 14:01:54.018409 2194 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.018811 2194 tablet_bootstrap.cc:492] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:54.019104 2194 ts_tablet_manager.cc:1367] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.044s user 0.028s sys 0.012s
I20240501 14:01:54.019730 2194 raft_consensus.cc:348] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:54.019961 2194 raft_consensus.cc:729] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:54.020156 2194 consensus_queue.cc:260] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:54.020637 2194 ts_tablet_manager.cc:1398] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:54.021494 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-5c35ccca22fb45b78ee66991027c0261 received from {username='slave'} at 127.22.5.198:37341
I20240501 14:01:54.021670 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-5c35ccca22fb45b78ee66991027c0261 on tablet 5c35ccca22fb45b78ee66991027c0261 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:54.025876 31946 raft_consensus.cc:1203] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.032948 2219 tablet_copy_client.cc:523] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.036185 2219 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:54.038684 32075 raft_consensus.cc:1203] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.040865 32075 raft_consensus.cc:1203] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.047387 575 raft_consensus.cc:1203] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.053628 2205 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.054148 2205 tablet_bootstrap.cc:492] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Bootstrap complete.
I20240501 14:01:54.054430 2205 ts_tablet_manager.cc:1367] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.008s
I20240501 14:01:54.055182 2205 raft_consensus.cc:348] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:54.055400 2205 raft_consensus.cc:729] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Initialized, Role: LEARNER
I20240501 14:01:54.055583 2205 consensus_queue.cc:260] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: true } }
I20240501 14:01:54.056114 2205 ts_tablet_manager.cc:1398] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:54.057097 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session 3321c47e58ba49f09f4f036ac72fc133-594dcb0a05054f2abaa11e5d4537b724 received from {username='slave'} at 127.22.5.198:37341
I20240501 14:01:54.057291 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session 3321c47e58ba49f09f4f036ac72fc133-594dcb0a05054f2abaa11e5d4537b724 on tablet 594dcb0a05054f2abaa11e5d4537b724 with peer 3321c47e58ba49f09f4f036ac72fc133
I20240501 14:01:54.076718 2219 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.077102 2219 tablet_bootstrap.cc:492] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:54.077383 2219 ts_tablet_manager.cc:1367] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.006s
I20240501 14:01:54.078011 2219 raft_consensus.cc:348] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:54.078235 2219 raft_consensus.cc:729] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:54.078382 2219 consensus_queue.cc:260] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:54.078845 2219 ts_tablet_manager.cc:1398] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:54.079691 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-0ea9aaebd437424eb27c04bc2f6ffa7b received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:54.079833 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-0ea9aaebd437424eb27c04bc2f6ffa7b on tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with peer aa11715897034babb1009134d8914066
I20240501 14:01:54.105243 1922 raft_consensus.cc:1050] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:54.105998 1922 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.108580 30031 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:54.108609 30291 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:54.108664 575 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:54.109323 1922 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:54.109994 2542 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:54.110589 32513 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:54.119985 2543 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.121415 30291 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.123178 575 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.124303 30031 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.126068 29752 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 4 to 5, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:54.128559 32210 raft_consensus.cc:1203] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.155282 31945 raft_consensus.cc:1203] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.182449 575 raft_consensus.cc:1203] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.189931 2266 tablet_copy_client.cc:523] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:54.193094 2266 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:54.202472 2185 raft_consensus.cc:1050] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:54.203383 2185 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:54.212740 31812 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.213502 2370 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.220360 32210 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.221150 2370 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.229646 30031 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.230407 2370 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.233722 2370 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.242076 30031 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.246949 575 raft_consensus.cc:1203] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.248390 29756 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.252138 32209 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.253989 31812 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.259310 2266 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:54.259795 2266 tablet_bootstrap.cc:492] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:54.260176 2266 ts_tablet_manager.cc:1367] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.067s user 0.053s sys 0.005s
I20240501 14:01:54.261873 2266 raft_consensus.cc:348] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:54.262178 2266 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:54.262380 2266 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:54.263113 2266 ts_tablet_manager.cc:1398] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:54.263787 30052 tablet_copy_service.cc:342] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Request end of tablet copy session aa11715897034babb1009134d8914066-c00649352f9d41c3a12a5e767b8b2aa6 received from {username='slave'} at 127.22.5.201:50499
I20240501 14:01:54.263985 30052 tablet_copy_service.cc:433] P 7ac7d9aa0b5e49de9962e9067d2bc0e4: ending tablet copy session aa11715897034babb1009134d8914066-c00649352f9d41c3a12a5e767b8b2aa6 on tablet c00649352f9d41c3a12a5e767b8b2aa6 with peer aa11715897034babb1009134d8914066
I20240501 14:01:54.279570 30291 consensus_queue.cc:237] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:54.280539 31945 raft_consensus.cc:1203] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:01:54.282054 32210 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.282248 31812 raft_consensus.cc:1261] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.283010 2370 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.283902 2370 consensus_queue.cc:1035] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.285931 2568 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.286562 32210 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.287426 31812 raft_consensus.cc:2932] T 9b56df9dcda34e6a8d27a8a6b24742b2 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.290220 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.293047 29756 catalog_manager.cc:5484] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 744cef852ee9491db3928fa0092ef189 reported cstate change: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:54.298718 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b56df9dcda34e6a8d27a8a6b24742b2 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:54.299413 2574 tablet_replica.cc:330] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:54.299626 2574 raft_consensus.cc:2229] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.300020 2574 raft_consensus.cc:2258] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.300891 2574 ts_tablet_manager.cc:1875] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.306941 2574 ts_tablet_manager.cc:1888] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.307191 2574 log.cc:1196] T 9b56df9dcda34e6a8d27a8a6b24742b2 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:01:54.308259 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 9b56df9dcda34e6a8d27a8a6b24742b2 (table rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]) successfully deleted
I20240501 14:01:54.326359 31812 raft_consensus.cc:1203] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.391259 32210 raft_consensus.cc:1203] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:01:54.392975 31945 raft_consensus.cc:1203] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.445902 29899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35082
I20240501 14:01:54.446158 29899 raft_consensus.cc:595] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:54.506430 1864 raft_consensus.cc:1050] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:54.507516 1864 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.510306 575 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.510546 31945 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.510939 30162 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.511029 2672 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.511672 2672 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.512200 2672 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.516134 2672 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.517202 31945 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.517807 30162 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.519922 575 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.521425 29757 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:54.528832 30031 consensus_queue.cc:237] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.531026 575 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.531123 31945 raft_consensus.cc:1261] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.532318 2674 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.532717 2672 consensus_queue.cc:1035] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.535667 1930 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.536312 31945 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.540155 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0929994d486d4610b9cc76abd39f043a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.540940 573 raft_consensus.cc:2932] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.545145 29753 catalog_manager.cc:5484] T 0929994d486d4610b9cc76abd39f043a P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.549364 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 0929994d486d4610b9cc76abd39f043a with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:54.550185 2688 tablet_replica.cc:330] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:54.550400 2688 raft_consensus.cc:2229] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.550652 2688 raft_consensus.cc:2258] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.551923 2688 ts_tablet_manager.cc:1875] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.557988 2688 ts_tablet_manager.cc:1888] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.558214 2688 log.cc:1196] T 0929994d486d4610b9cc76abd39f043a P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/0929994d486d4610b9cc76abd39f043a
I20240501 14:01:54.559235 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 0929994d486d4610b9cc76abd39f043a (table rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]) successfully deleted
I20240501 14:01:54.568384 32513 raft_consensus.cc:979] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: : Instructing follower 4a15e881772b41eab8a71fc027390db9 to start an election
I20240501 14:01:54.568567 32513 raft_consensus.cc:1067] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Signalling peer 4a15e881772b41eab8a71fc027390db9 to start an election
I20240501 14:01:54.569396 30291 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0"
dest_uuid: "4a15e881772b41eab8a71fc027390db9"
from {username='slave'} at 127.22.5.193:32853
I20240501 14:01:54.569649 30291 raft_consensus.cc:482] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:54.569782 30291 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:54.572636 30291 raft_consensus.cc:504] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.574232 29899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:54.574232 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:54.574484 30031 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:54.574484 29899 raft_consensus.cc:3032] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:54.574573 29899 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:54.574932 29899 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.575353 29899 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:54.575369 30291 leader_election.cc:288] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:01:54.577622 30031 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:54.579341 29899 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:54.579933 30225 leader_election.cc:302] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:54.580302 2568 raft_consensus.cc:2781] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:54.580473 2568 raft_consensus.cc:686] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:54.580796 2568 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.584833 29757 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 1 to 2, leader changed from 7869901e77234aeeb1462876b63eb28e (127.22.5.193) to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.586894 573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6d21ea3184a6eb9db2b87589781b0" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "aa11715897034babb1009134d8914066"
I20240501 14:01:54.587241 573 raft_consensus.cc:3037] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:54.590155 573 raft_consensus.cc:2454] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:54.610525 1922 raft_consensus.cc:1050] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:54.611393 1922 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:54.613996 32075 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.614120 32210 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.614147 30031 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.614742 2543 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.615100 1922 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.615371 2717 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.618443 1930 raft_consensus.cc:1050] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:54.619227 1930 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.627184 2185 raft_consensus.cc:1050] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:01:54.628018 2185 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:54.631134 31812 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.631778 29899 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.632409 1922 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.633045 2370 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.634310 1864 raft_consensus.cc:1050] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:54.634831 32513 raft_consensus.cc:1050] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:54.635198 1864 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:54.635378 32075 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.635646 32513 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.637663 32076 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.637670 30031 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.638199 2568 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.640400 2370 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.641716 31945 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.642397 2672 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.644389 29756 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.645447 2568 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.651556 32076 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.651677 30291 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.651635 29899 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.652393 2672 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.652743 1930 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.657829 31812 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.659238 29757 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.668929 32077 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.669443 2566 raft_consensus.cc:1050] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:54.670548 2566 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.674232 32209 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.675769 31946 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.675719 2672 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.676672 573 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.676347 2672 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.677386 2717 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.680487 31946 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.681574 31946 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.682348 2717 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.682296 29753 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.682714 2566 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 14:01:54.683833 30162 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.683792 2674 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.684839 30031 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.685688 2717 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.685343 30291 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.688851 30290 consensus_queue.cc:237] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:54.690846 32076 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.691125 2568 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.694132 2568 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.698843 31812 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.699685 2370 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.704021 32077 raft_consensus.cc:1261] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.703696 2568 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.704753 2370 consensus_queue.cc:1035] T 56f632889b1e400392fe710bf9f6f3ec P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.705826 30291 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.706317 2674 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.706437 32077 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.707973 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56f632889b1e400392fe710bf9f6f3ec with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.708698 31813 raft_consensus.cc:2932] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.709177 2370 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.709851 30031 consensus_queue.cc:237] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.712718 31945 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.713804 31946 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.714917 2543 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.718276 31942 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.719544 573 raft_consensus.cc:1203] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:54.721257 29757 catalog_manager.cc:5484] T 56f632889b1e400392fe710bf9f6f3ec P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 8 to 9, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.722965 30031 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.725775 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f632889b1e400392fe710bf9f6f3ec with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:54.729249 573 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.730433 29757 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.731247 31944 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.732026 1864 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.734428 32210 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:54.739084 1930 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.681434 32075 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.745071 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.749431 29899 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.750195 31944 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.752486 573 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.753181 2543 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.756099 31941 raft_consensus.cc:1261] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.757036 1922 consensus_queue.cc:1035] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.758559 31941 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.758361 29757 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.761502 2717 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.761940 29752 catalog_manager.cc:5484] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.767552 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 427af2a000f34ab6b9e3c9c7d7be1b32 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:54.771569 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 368edc181c894686a213fc542f328192 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.772954 2741 tablet_replica.cc:330] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:54.774040 573 raft_consensus.cc:2932] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.774619 29756 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:54.775280 2741 raft_consensus.cc:2229] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.775787 2741 raft_consensus.cc:2258] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.777735 2741 ts_tablet_manager.cc:1875] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.780653 2744 tablet_replica.cc:330] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:54.780995 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 368edc181c894686a213fc542f328192 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:54.784237 2744 raft_consensus.cc:2229] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.784685 2744 raft_consensus.cc:2258] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.787854 2744 ts_tablet_manager.cc:1875] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.799556 30290 consensus_queue.cc:237] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.799130 32076 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.801920 2744 ts_tablet_manager.cc:1888] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.802517 2744 log.cc:1196] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:01:54.803418 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 427af2a000f34ab6b9e3c9c7d7be1b32 (table rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]) successfully deleted
I20240501 14:01:54.804565 2543 raft_consensus.cc:1050] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:54.806053 32074 raft_consensus.cc:1261] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.805869 1864 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.807044 31941 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.807924 2568 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.808195 2688 tablet_replica.cc:330] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:54.808399 2688 raft_consensus.cc:2229] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.808609 2688 pending_rounds.cc:62] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Trying to abort 1 pending ops.
I20240501 14:01:54.808717 2688 pending_rounds.cc:68] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022887381545508864 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "368edc181c894686a213fc542f328192" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } } }
I20240501 14:01:54.809219 2688 raft_consensus.cc:2894] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 14:01:54.809218 2672 consensus_queue.cc:1035] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.809351 2688 raft_consensus.cc:2866] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:01:54.809466 2688 raft_consensus.cc:2258] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.810354 2688 ts_tablet_manager.cc:1875] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.811192 32209 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.811803 2543 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.813321 2566 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.813867 31941 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.817354 32075 raft_consensus.cc:2932] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.819869 2688 ts_tablet_manager.cc:1888] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.822753 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c40a79b118424a5699bddd1c4ac4bb0e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.823642 29752 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 reported cstate change: config changed from index 7 to 8, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.823742 30290 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.824390 32073 raft_consensus.cc:1261] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.824163 31941 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.825273 2566 consensus_queue.cc:1035] T c40a79b118424a5699bddd1c4ac4bb0e P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.826750 32073 raft_consensus.cc:2932] T c40a79b118424a5699bddd1c4ac4bb0e P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.833863 2688 log.cc:1196] T 368edc181c894686a213fc542f328192 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/368edc181c894686a213fc542f328192
I20240501 14:01:54.835623 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 368edc181c894686a213fc542f328192 (table rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]) successfully deleted
I20240501 14:01:54.835465 29756 catalog_manager.cc:5484] T c40a79b118424a5699bddd1c4ac4bb0e P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.838542 30031 consensus_queue.cc:237] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:54.841727 31943 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.842048 32210 raft_consensus.cc:1261] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:54.842429 2672 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.843401 1930 consensus_queue.cc:1035] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.846103 1922 raft_consensus.cc:1050] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:54.846935 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet c40a79b118424a5699bddd1c4ac4bb0e with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:54.846925 1922 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:54.852401 1930 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.857792 31943 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.858680 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4988f0de3214860a65007b8e31e88a5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:54.861759 29753 catalog_manager.cc:5484] T f4988f0de3214860a65007b8e31e88a5 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:54.864190 2717 raft_consensus.cc:1050] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER 3321c47e58ba49f09f4f036ac72fc133 to VOTER
I20240501 14:01:54.865007 2717 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:54.866833 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet f4988f0de3214860a65007b8e31e88a5 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:54.869695 31945 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:54.870220 32513 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.874176 2744 tablet_replica.cc:330] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:54.874459 2744 raft_consensus.cc:2229] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.874950 2744 raft_consensus.cc:2258] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.876262 2744 ts_tablet_manager.cc:1875] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.884493 32209 raft_consensus.cc:2932] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:54.890987 573 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.891201 2749 tablet_replica.cc:330] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:54.891628 2749 raft_consensus.cc:2229] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:54.891283 30031 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.892035 2749 raft_consensus.cc:2258] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:54.893265 31941 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.893656 2749 ts_tablet_manager.cc:1875] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:54.893905 32513 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.894798 2741 ts_tablet_manager.cc:1888] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.894953 2741 log.cc:1196] T 56f632889b1e400392fe710bf9f6f3ec P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/56f632889b1e400392fe710bf9f6f3ec
I20240501 14:01:54.898241 32513 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.908867 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 56f632889b1e400392fe710bf9f6f3ec (table rebalance_test_table_7 [id=bfbe925346fc4f05bc7d0db8e253d01d]) successfully deleted
I20240501 14:01:54.916592 2747 raft_consensus.cc:1050] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER be55dad80e3a4ad79d21aaa3199b1d0b to VOTER
I20240501 14:01:54.926683 2749 ts_tablet_manager.cc:1888] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.927067 32513 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.927783 2744 ts_tablet_manager.cc:1888] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:54.927842 30290 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:54.928712 2744 log.cc:1196] T f4988f0de3214860a65007b8e31e88a5 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4988f0de3214860a65007b8e31e88a5
I20240501 14:01:54.929746 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet f4988f0de3214860a65007b8e31e88a5 (table rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]) successfully deleted
I20240501 14:01:54.931149 2749 log.cc:1196] T c40a79b118424a5699bddd1c4ac4bb0e P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:01:54.931936 1922 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.932066 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet c40a79b118424a5699bddd1c4ac4bb0e (table rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]) successfully deleted
I20240501 14:01:54.933547 2758 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.933591 573 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.935382 31813 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:54.936177 2754 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:54.939389 2753 raft_consensus.cc:1050] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565: attempting to promote NON_VOTER 744cef852ee9491db3928fa0092ef189 to VOTER
I20240501 14:01:54.941483 29752 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:54.941967 31943 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.940827 30031 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.943146 2753 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:54.945384 2717 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.949223 2747 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:54.950019 30162 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.951233 2543 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.953177 32209 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.955737 2754 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.969332 31812 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.969097 31945 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:54.970185 2573 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.970139 31941 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.971431 2573 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.974242 30291 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:54.974856 2761 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:54.981572 2717 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.983968 31943 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.986598 30162 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:54.992982 29752 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:55.003372 30031 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:55.004184 2370 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:55.011014 573 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:55.011734 2370 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:55.047559 29899 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:01:55.048250 2568 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:55.048892 32210 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 LEARNER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:55.048678 32209 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.050868 30292 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:55.050869 29899 consensus_queue.cc:237] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:55.051822 29900 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:55.053077 31945 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.053702 2758 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.053867 2717 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.056860 2782 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.057835 2311 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.057616 31813 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.058574 2311 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.061556 2754 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.064810 2761 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.066200 31945 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.068254 31943 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.069448 31812 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.080003 29753 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.080648 30292 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.086750 29753 catalog_manager.cc:5484] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.087492 29757 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 8 to 9, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.094266 30291 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.100486 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c35ccca22fb45b78ee66991027c0261 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.102768 31943 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.111096 573 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:55.111814 2566 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.109048 30290 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.113391 32210 raft_consensus.cc:1261] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.114367 2717 consensus_queue.cc:1035] T 5c35ccca22fb45b78ee66991027c0261 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.117079 2740 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.117769 575 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.121850 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2b6d21ea3184a6eb9db2b87589781b0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.123386 32209 raft_consensus.cc:2932] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.124321 29752 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 reported cstate change: config changed from index 5 to 7, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.131829 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b6d21ea3184a6eb9db2b87589781b0 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:55.139371 2741 tablet_replica.cc:330] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:55.139667 2741 raft_consensus.cc:2229] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus shutting down.
W20240501 14:01:55.139624 30095 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:34471, user_credentials={real_user=slave}} blocked reactor thread for 63459.2us
I20240501 14:01:55.139889 2741 raft_consensus.cc:2258] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.140254 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c35ccca22fb45b78ee66991027c0261 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:55.140626 2688 tablet_replica.cc:330] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:55.140885 2688 raft_consensus.cc:2229] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.141330 2688 raft_consensus.cc:2258] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.141547 2741 ts_tablet_manager.cc:1875] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.142550 2688 ts_tablet_manager.cc:1875] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.141973 2311 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.147387 30290 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.151355 2688 ts_tablet_manager.cc:1888] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:55.151573 2688 log.cc:1196] T 5c35ccca22fb45b78ee66991027c0261 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c35ccca22fb45b78ee66991027c0261
I20240501 14:01:55.153286 29901 consensus_queue.cc:237] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:01:55.158308 31813 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:55.158716 31943 raft_consensus.cc:1261] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:55.159040 2717 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:55.159397 2754 consensus_queue.cc:1035] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:55.161899 2760 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 7869901e77234aeeb1462876b63eb28e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.163697 31813 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.165050 2741 ts_tablet_manager.cc:1888] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:01:55.167330 31943 raft_consensus.cc:2932] T 594dcb0a05054f2abaa11e5d4537b724 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.168188 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 5c35ccca22fb45b78ee66991027c0261 (table rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]) successfully deleted
I20240501 14:01:55.169883 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 594dcb0a05054f2abaa11e5d4537b724 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.171388 29752 catalog_manager.cc:5484] T 594dcb0a05054f2abaa11e5d4537b724 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 9 to 10, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 2 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } }
I20240501 14:01:55.172612 30031 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:01:55.175770 2740 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.178016 30032 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.178750 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 594dcb0a05054f2abaa11e5d4537b724 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:55.179276 2744 tablet_replica.cc:330] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:55.179472 2744 raft_consensus.cc:2229] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.179685 2744 raft_consensus.cc:2258] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.180684 2744 ts_tablet_manager.cc:1875] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.180680 29752 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 8 to 9, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.184897 2741 log.cc:1196] T f2b6d21ea3184a6eb9db2b87589781b0 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:01:55.186865 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet f2b6d21ea3184a6eb9db2b87589781b0 (table rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]) successfully deleted
I20240501 14:01:55.188194 32209 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.188390 30162 consensus_queue.cc:237] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } }
I20240501 14:01:55.178406 29901 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 744cef852ee9491db3928fa0092ef189 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.191341 31813 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.192014 2573 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.193904 31941 raft_consensus.cc:1261] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.194514 2785 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.195142 31813 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.194522 2573 consensus_queue.cc:1035] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240501 14:01:55.201391 29757 catalog_manager.cc:5484] T 98d94ed11f0f4a9d9f519a24c3320ce4 P be55dad80e3a4ad79d21aaa3199b1d0b reported cstate change: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.203159 31941 raft_consensus.cc:2932] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } }
I20240501 14:01:55.204957 2744 ts_tablet_manager.cc:1888] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:55.205088 2744 log.cc:1196] T 594dcb0a05054f2abaa11e5d4537b724 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/594dcb0a05054f2abaa11e5d4537b724
I20240501 14:01:55.207058 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 594dcb0a05054f2abaa11e5d4537b724 (table rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]) successfully deleted
I20240501 14:01:55.208357 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:55.208920 2744 tablet_replica.cc:330] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:55.209162 2744 raft_consensus.cc:2229] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.208853 32075 consensus_queue.cc:237] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:01:55.209517 2744 raft_consensus.cc:2258] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.210458 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98d94ed11f0f4a9d9f519a24c3320ce4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.214062 32209 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:55.214052 30290 raft_consensus.cc:1261] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:55.214964 2753 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:55.215431 2782 consensus_queue.cc:1035] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:55.219732 2778 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.224895 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b1ada0fe1ce431a823f2814f116db28 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.226400 30290 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.228070 29752 catalog_manager.cc:5484] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: config changed from index 9 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:55.240228 2744 ts_tablet_manager.cc:1875] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.241958 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1ada0fe1ce431a823f2814f116db28 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:55.242326 2741 tablet_replica.cc:330] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:55.242527 2741 raft_consensus.cc:2229] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.242987 2741 raft_consensus.cc:2258] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.244010 2741 ts_tablet_manager.cc:1875] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.246205 32210 raft_consensus.cc:2932] T 2b1ada0fe1ce431a823f2814f116db28 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } }
I20240501 14:01:55.251344 2672 raft_consensus.cc:1050] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:55.252491 2672 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.256007 2741 ts_tablet_manager.cc:1888] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:55.256186 2741 log.cc:1196] T 2b1ada0fe1ce431a823f2814f116db28 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b1ada0fe1ce431a823f2814f116db28
I20240501 14:01:55.259671 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 2b1ada0fe1ce431a823f2814f116db28 (table rebalance_test_table_4 [id=ed9c8c2f29f84f6d94f70df08092ed1e]) successfully deleted
I20240501 14:01:55.260812 2744 ts_tablet_manager.cc:1888] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:55.260970 2744 log.cc:1196] T 98d94ed11f0f4a9d9f519a24c3320ce4 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:01:55.262138 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 98d94ed11f0f4a9d9f519a24c3320ce4 (table rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]) successfully deleted
I20240501 14:01:55.265822 31813 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:55.265785 573 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:55.266201 29901 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Refusing update from remote peer 7ac7d9aa0b5e49de9962e9067d2bc0e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:55.266968 1930 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:55.267323 1864 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:55.268888 29901 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:55.267867 2672 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:55.277758 1864 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.279660 30162 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:55.280200 29835 consensus_peers.cc:485] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 212e139f1e7c4674987c2309efe3e495. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:55.280313 2717 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.280216 29901 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.281395 31813 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:55.282097 31812 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.282644 573 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.283860 2717 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.288525 29752 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:55.295064 2760 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.295894 30162 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.303735 29757 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.305146 31813 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.342746 30032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40322
I20240501 14:01:55.342981 30032 raft_consensus.cc:595] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:55.361665 29901 consensus_queue.cc:237] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.364399 573 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.364506 31941 raft_consensus.cc:1261] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:55.365155 2754 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.365473 2717 consensus_queue.cc:1035] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:55.369313 2754 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.371184 31941 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.372942 573 raft_consensus.cc:2932] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.376502 29753 catalog_manager.cc:5484] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: config changed from index 8 to 9, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:55.380847 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:55.383419 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea9aaebd437424eb27c04bc2f6ffa7b with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:55.383848 2749 tablet_replica.cc:330] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:55.384075 2749 raft_consensus.cc:2229] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:55.384312 2749 raft_consensus.cc:2258] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:55.385075 2749 ts_tablet_manager.cc:1875] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:55.390864 2749 ts_tablet_manager.cc:1888] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:55.390990 2749 log.cc:1196] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:01:55.391780 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet 0ea9aaebd437424eb27c04bc2f6ffa7b (table rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]) successfully deleted
I20240501 14:01:55.496172 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35190
I20240501 14:01:55.496380 29901 raft_consensus.cc:595] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:55.517872 30162 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:55.520362 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:55.520876 32209 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:55.521200 2787 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.521553 2785 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.524143 2787 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.528362 30290 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
W20240501 14:01:55.529449 30096 consensus_peers.cc:485] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 642312cdad114fce839d2b60717bcc72. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:55.532927 29753 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:55.539577 32209 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.557041 30290 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:55.559541 30162 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:55.559655 30226 consensus_peers.cc:485] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7d3626b1c8d458e9c6d4a28bc0564fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:55.559691 32075 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:55.560257 2740 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.560544 2568 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.563185 2740 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.564114 32075 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.564471 30162 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.570161 29753 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.582136 29901 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:55.584338 32209 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:55.584504 30290 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:55.584553 29835 consensus_peers.cc:485] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 582bddb40ece4d5b8dcd3e1c00a53536. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:55.584957 2760 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.585345 2754 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:55.588017 2717 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.588588 32209 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.588779 30290 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:55.592839 29757 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:55.607412 2758 raft_consensus.cc:979] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: : Instructing follower 47b7351d2e9b4dbd9a09fd4133a97565 to start an election
I20240501 14:01:55.607595 2758 raft_consensus.cc:1067] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Signalling peer 47b7351d2e9b4dbd9a09fd4133a97565 to start an election
I20240501 14:01:55.608292 32075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1"
dest_uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
from {username='slave'} at 127.22.5.193:35815
I20240501 14:01:55.608526 32075 raft_consensus.cc:482] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:55.608642 32075 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.611680 32075 raft_consensus.cc:504] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:55.612519 32075 leader_election.cc:288] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:01:55.612920 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:55.613325 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:55.613391 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31303ad8c9124217a45e26f8da5cc0a1" candidate_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "744cef852ee9491db3928fa0092ef189"
I20240501 14:01:55.613569 30032 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.613595 32209 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.616555 30032 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:55.616822 32209 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:55.616943 29901 raft_consensus.cc:3032] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:55.617072 29901 raft_consensus.cc:729] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:55.617393 29901 consensus_queue.cc:260] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:55.617692 32013 leader_election.cc:302] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47b7351d2e9b4dbd9a09fd4133a97565, 744cef852ee9491db3928fa0092ef189, 7ac7d9aa0b5e49de9962e9067d2bc0e4; no voters:
I20240501 14:01:55.617959 29901 raft_consensus.cc:3037] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.618175 2793 raft_consensus.cc:2781] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:55.618319 2793 raft_consensus.cc:686] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Becoming Leader. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Running, Role: LEADER
I20240501 14:01:55.618641 2793 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:55.620586 29901 raft_consensus.cc:2454] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b7351d2e9b4dbd9a09fd4133a97565 in term 2.
I20240501 14:01:55.622054 29756 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: term changed from 1 to 2, leader changed from 7869901e77234aeeb1462876b63eb28e (127.22.5.193) to 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199). New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:55.671332 30032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6"
dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40354
I20240501 14:01:55.671603 30032 raft_consensus.cc:595] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:55.719403 1930 raft_consensus.cc:979] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: : Instructing follower aa11715897034babb1009134d8914066 to start an election
I20240501 14:01:55.719632 1930 raft_consensus.cc:1067] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Signalling peer aa11715897034babb1009134d8914066 to start an election
I20240501 14:01:55.720455 573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6"
dest_uuid: "aa11715897034babb1009134d8914066"
from {username='slave'} at 127.22.5.194:38879
I20240501 14:01:55.720762 573 raft_consensus.cc:482] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:55.720878 573 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.722841 573 raft_consensus.cc:504] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.724404 573 leader_election.cc:288] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
I20240501 14:01:55.724835 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:55.724872 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:01:55.725088 29901 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.725102 30032 raft_consensus.cc:3032] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:55.725239 30032 raft_consensus.cc:729] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ac7d9aa0b5e49de9962e9067d2bc0e4, State: Running, Role: LEADER
I20240501 14:01:55.725509 30032 consensus_queue.cc:260] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.725944 30032 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.730132 29901 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:01:55.730772 30032 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:01:55.731520 479 leader_election.cc:302] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4, aa11715897034babb1009134d8914066; no voters:
I20240501 14:01:55.731993 2449 raft_consensus.cc:2781] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:55.732517 2449 raft_consensus.cc:686] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 2 LEADER]: Becoming Leader. State: Replica: aa11715897034babb1009134d8914066, State: Running, Role: LEADER
I20240501 14:01:55.733084 2449 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:55.737974 31813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00649352f9d41c3a12a5e767b8b2aa6" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b"
I20240501 14:01:55.738250 31813 raft_consensus.cc:3037] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:55.738409 29753 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 reported cstate change: term changed from 1 to 2, leader changed from 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) to aa11715897034babb1009134d8914066 (127.22.5.201). New cstate: current_term: 2 leader_uuid: "aa11715897034babb1009134d8914066" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:55.740557 31813 raft_consensus.cc:2454] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:01:55.825464 2943 ts_tablet_manager.cc:902] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:55.832019 2943 tablet_copy_client.cc:308] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:55.832821 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 212e139f1e7c4674987c2309efe3e495 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:55.833010 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 212e139f1e7c4674987c2309efe3e495 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-212e139f1e7c4674987c2309efe3e495
I20240501 14:01:55.835494 29921 tablet_copy_source_session.cc:205] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:55.836750 2943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212e139f1e7c4674987c2309efe3e495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:55.842417 2943 tablet_copy_client.cc:783] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:55.842706 2943 tablet_copy_client.cc:647] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:55.844245 2947 tablet_copy_client.cc:1002] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:55.926180 2963 ts_tablet_manager.cc:902] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823)
I20240501 14:01:55.927685 2963 tablet_copy_client.cc:308] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:38823
I20240501 14:01:55.933969 30182 tablet_copy_service.cc:140] P 76c7a691de8a4e31b32799f43cb4b812: Received BeginTabletCopySession request for tablet 642312cdad114fce839d2b60717bcc72 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:55953)
I20240501 14:01:55.934124 30182 tablet_copy_service.cc:161] P 76c7a691de8a4e31b32799f43cb4b812: Beginning new tablet copy session on tablet 642312cdad114fce839d2b60717bcc72 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:55953: session id = aa11715897034babb1009134d8914066-642312cdad114fce839d2b60717bcc72
I20240501 14:01:55.936858 30182 tablet_copy_source_session.cc:205] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:55.938266 2963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642312cdad114fce839d2b60717bcc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:55.943940 2963 tablet_copy_client.cc:783] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:55.944339 2963 tablet_copy_client.cc:647] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:55.946223 2967 tablet_copy_client.cc:1002] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:56.091466 32209 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.092094 2311 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.108770 2985 ts_tablet_manager.cc:902] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:56.110323 2985 tablet_copy_client.cc:308] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:56.111079 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet 582bddb40ece4d5b8dcd3e1c00a53536 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:56.111227 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet 582bddb40ece4d5b8dcd3e1c00a53536 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:01:56.113169 29921 tablet_copy_source_session.cc:205] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:56.114367 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582bddb40ece4d5b8dcd3e1c00a53536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:56.118757 2985 tablet_copy_client.cc:783] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:56.119006 2985 tablet_copy_client.cc:647] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:56.120538 2989 tablet_copy_client.cc:1002] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:56.125584 29901 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.126215 2449 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.138567 2990 ts_tablet_manager.cc:902] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:01:56.139824 2990 tablet_copy_client.cc:308] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:01:56.145188 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet f7d3626b1c8d458e9c6d4a28bc0564fa from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:46405)
I20240501 14:01:56.145373 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet f7d3626b1c8d458e9c6d4a28bc0564fa from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:46405: session id = aa11715897034babb1009134d8914066-f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:01:56.147408 30312 tablet_copy_source_session.cc:205] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:56.148794 2990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3626b1c8d458e9c6d4a28bc0564fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:56.154387 2990 tablet_copy_client.cc:783] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:56.154641 2990 tablet_copy_client.cc:647] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:56.156286 2994 tablet_copy_client.cc:1002] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:56.162340 29901 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.162899 2311 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.188673 30032 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.189525 2996 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.216635 30032 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.217301 2449 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.226411 31813 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:56.227203 2942 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:56.600642 32075 consensus_queue.cc:237] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:01:56.602542 32210 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:56.602566 30032 raft_consensus.cc:1261] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 47b7351d2e9b4dbd9a09fd4133a97565: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:56.603104 3002 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:56.603618 3002 consensus_queue.cc:1035] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:56.605659 2996 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:56.606443 30032 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:56.606405 32210 raft_consensus.cc:2932] T 31303ad8c9124217a45e26f8da5cc0a1 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:56.610723 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31303ad8c9124217a45e26f8da5cc0a1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:56.611390 29756 catalog_manager.cc:5484] T 31303ad8c9124217a45e26f8da5cc0a1 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 reported cstate change: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:01:56.615653 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 31303ad8c9124217a45e26f8da5cc0a1 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:56.616477 3076 tablet_replica.cc:330] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:56.616696 3076 raft_consensus.cc:2229] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:56.616925 3076 raft_consensus.cc:2258] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:56.618111 3076 ts_tablet_manager.cc:1875] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:56.624878 3076 ts_tablet_manager.cc:1888] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:56.625025 3076 log.cc:1196] T 31303ad8c9124217a45e26f8da5cc0a1 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:01:56.633087 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 31303ad8c9124217a45e26f8da5cc0a1 (table rebalance_test_table_6 [id=83ebcb0883f845c790173a952765434d]) successfully deleted
I20240501 14:01:56.793215 573 consensus_queue.cc:237] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:56.795379 29901 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:56.795553 31813 raft_consensus.cc:1261] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:56.796067 3082 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:56.796413 3083 consensus_queue.cc:1035] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:56.799327 3082 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:56.800169 29901 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:56.804960 29745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c00649352f9d41c3a12a5e767b8b2aa6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:56.805850 29756 catalog_manager.cc:5484] T c00649352f9d41c3a12a5e767b8b2aa6 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 8 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 2 leader_uuid: "aa11715897034babb1009134d8914066" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:56.807684 31813 raft_consensus.cc:2932] T c00649352f9d41c3a12a5e767b8b2aa6 P be55dad80e3a4ad79d21aaa3199b1d0b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:56.809986 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet c00649352f9d41c3a12a5e767b8b2aa6 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37382
I20240501 14:01:56.810647 3096 tablet_replica.cc:330] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:01:56.810930 3096 raft_consensus.cc:2229] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:56.811192 3096 raft_consensus.cc:2258] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:56.813867 3096 ts_tablet_manager.cc:1875] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:56.821494 3096 ts_tablet_manager.cc:1888] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:56.821830 3096 log.cc:1196] T c00649352f9d41c3a12a5e767b8b2aa6 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:01:56.823118 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet c00649352f9d41c3a12a5e767b8b2aa6 (table rebalance_test_table_5 [id=44a56685fda143629c6f0fd70a902508]) successfully deleted
I20240501 14:01:56.874907 29901 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:56.877844 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:56.878201 32075 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:56.878945 2717 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:56.879271 2758 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:56.879595 29835 consensus_peers.cc:485] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e -> Peer aa11715897034babb1009134d8914066 (127.22.5.201:38269): Couldn't send request to peer aa11715897034babb1009134d8914066. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef202920b8d64287a8a2cdb40c73a819. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:56.882326 2758 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:56.883112 32075 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:56.888697 29757 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:56.888783 30032 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa11715897034babb1009134d8914066 (127.22.5.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } } }
I20240501 14:01:57.344655 2943 tablet_copy_client.cc:523] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:57.348506 2943 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:57.385378 2943 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:57.386451 2943 tablet_bootstrap.cc:492] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:57.387223 2943 ts_tablet_manager.cc:1367] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.039s user 0.040s sys 0.000s
I20240501 14:01:57.388103 2943 raft_consensus.cc:348] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.388361 2943 raft_consensus.cc:729] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:57.388556 2943 consensus_queue.cc:260] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.389098 2943 ts_tablet_manager.cc:1398] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:57.389943 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-212e139f1e7c4674987c2309efe3e495 received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:57.390105 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-212e139f1e7c4674987c2309efe3e495 on tablet 212e139f1e7c4674987c2309efe3e495 with peer aa11715897034babb1009134d8914066
I20240501 14:01:57.446677 2963 tablet_copy_client.cc:523] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:57.450012 2963 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:57.471801 2943 ts_tablet_manager.cc:902] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Initiating tablet copy from peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665)
I20240501 14:01:57.476871 2943 tablet_copy_client.cc:308] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:35665
I20240501 14:01:57.477779 29921 tablet_copy_service.cc:140] P 7869901e77234aeeb1462876b63eb28e: Received BeginTabletCopySession request for tablet ef202920b8d64287a8a2cdb40c73a819 from peer aa11715897034babb1009134d8914066 ({username='slave'} at 127.22.5.201:54107)
I20240501 14:01:57.477969 29921 tablet_copy_service.cc:161] P 7869901e77234aeeb1462876b63eb28e: Beginning new tablet copy session on tablet ef202920b8d64287a8a2cdb40c73a819 from peer aa11715897034babb1009134d8914066 at {username='slave'} at 127.22.5.201:54107: session id = aa11715897034babb1009134d8914066-ef202920b8d64287a8a2cdb40c73a819
I20240501 14:01:57.480103 29921 tablet_copy_source_session.cc:205] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:57.481305 2943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef202920b8d64287a8a2cdb40c73a819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:57.484483 2963 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:57.484920 2963 tablet_bootstrap.cc:492] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:57.485199 2963 ts_tablet_manager.cc:1367] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.009s
I20240501 14:01:57.485973 2943 tablet_copy_client.cc:783] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:57.485863 2963 raft_consensus.cc:348] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.486153 2963 raft_consensus.cc:729] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:57.486197 2943 tablet_copy_client.cc:647] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:57.486351 2963 consensus_queue.cc:260] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.486917 2963 ts_tablet_manager.cc:1398] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:57.487782 30182 tablet_copy_service.cc:342] P 76c7a691de8a4e31b32799f43cb4b812: Request end of tablet copy session aa11715897034babb1009134d8914066-642312cdad114fce839d2b60717bcc72 received from {username='slave'} at 127.22.5.201:55953
I20240501 14:01:57.487821 3202 tablet_copy_client.cc:1002] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: tablet copy: Injecting latency into file download: 1500
I20240501 14:01:57.487923 30182 tablet_copy_service.cc:433] P 76c7a691de8a4e31b32799f43cb4b812: ending tablet copy session aa11715897034babb1009134d8914066-642312cdad114fce839d2b60717bcc72 on tablet 642312cdad114fce839d2b60717bcc72 with peer aa11715897034babb1009134d8914066
I20240501 14:01:57.620939 2985 tablet_copy_client.cc:523] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:57.623569 2985 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:57.656687 2990 tablet_copy_client.cc:523] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:57.660574 2990 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:57.662042 2985 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:57.662514 2985 tablet_bootstrap.cc:492] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:57.662815 2985 ts_tablet_manager.cc:1367] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.039s user 0.034s sys 0.004s
I20240501 14:01:57.663722 2985 raft_consensus.cc:348] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.664054 2985 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:57.664283 2985 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.664920 2985 ts_tablet_manager.cc:1398] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:57.665745 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-582bddb40ece4d5b8dcd3e1c00a53536 received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:57.665886 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-582bddb40ece4d5b8dcd3e1c00a53536 on tablet 582bddb40ece4d5b8dcd3e1c00a53536 with peer aa11715897034babb1009134d8914066
I20240501 14:01:57.712787 2990 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:57.713306 2990 tablet_bootstrap.cc:492] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:57.713616 2990 ts_tablet_manager.cc:1367] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.053s user 0.046s sys 0.005s
I20240501 14:01:57.714241 2990 raft_consensus.cc:348] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.714454 2990 raft_consensus.cc:729] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:57.714643 2990 consensus_queue.cc:260] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:57.715129 2990 ts_tablet_manager.cc:1398] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:57.716059 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session aa11715897034babb1009134d8914066-f7d3626b1c8d458e9c6d4a28bc0564fa received from {username='slave'} at 127.22.5.201:46405
I20240501 14:01:57.716182 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session aa11715897034babb1009134d8914066-f7d3626b1c8d458e9c6d4a28bc0564fa on tablet f7d3626b1c8d458e9c6d4a28bc0564fa with peer aa11715897034babb1009134d8914066
I20240501 14:01:57.769172 573 raft_consensus.cc:1203] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:57.799249 573 raft_consensus.cc:1203] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:57.939247 573 raft_consensus.cc:1203] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:58.147619 2717 raft_consensus.cc:1050] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:58.148367 2717 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.151304 573 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.151533 575 raft_consensus.cc:1203] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:58.151841 30290 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.152884 2758 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.153528 2717 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.154835 32210 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.155759 2758 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.158326 2758 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.159121 30290 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.163398 575 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.165052 29753 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:58.167564 32210 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.222332 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56560
I20240501 14:01:58.222534 29901 raft_consensus.cc:595] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:58.234655 2717 raft_consensus.cc:1050] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:58.235554 2717 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.247377 575 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.247537 31813 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.248034 30162 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.248054 2717 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.248668 2717 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.249208 2717 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.253464 3258 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.254398 30162 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.259295 575 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.260787 29752 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:58.265201 31813 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.273021 29901 consensus_queue.cc:237] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.275475 575 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.276026 3258 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.276747 31812 raft_consensus.cc:1261] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.277439 3258 consensus_queue.cc:1035] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.279484 2758 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.281462 575 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.281958 31812 raft_consensus.cc:2932] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.285044 29757 catalog_manager.cc:5484] T 212e139f1e7c4674987c2309efe3e495 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:58.285362 29744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 212e139f1e7c4674987c2309efe3e495 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:58.290047 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet 212e139f1e7c4674987c2309efe3e495 with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:58.290835 3302 tablet_replica.cc:330] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:58.291235 3302 raft_consensus.cc:2229] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:58.291595 3302 raft_consensus.cc:2258] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:58.292563 3302 ts_tablet_manager.cc:1875] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:58.297674 3302 ts_tablet_manager.cc:1888] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:58.297809 3302 log.cc:1196] T 212e139f1e7c4674987c2309efe3e495 P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/212e139f1e7c4674987c2309efe3e495
I20240501 14:01:58.298660 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet 212e139f1e7c4674987c2309efe3e495 (table rebalance_test_table_9 [id=32564f7811fc4ec2adf8f248c02c3882]) successfully deleted
I20240501 14:01:58.326390 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56598
I20240501 14:01:58.326678 29901 raft_consensus.cc:595] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:58.330467 3318 raft_consensus.cc:1050] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:58.331334 3318 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.342608 32210 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.342743 575 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.343441 30290 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.343537 3039 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.345604 3319 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.347865 3039 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.351891 3039 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.357637 32210 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.362648 30290 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.363797 29757 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.376021 575 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.379433 30162 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.382205 575 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.382391 32210 raft_consensus.cc:1261] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Refusing update from remote peer 76c7a691de8a4e31b32799f43cb4b812: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.382804 3326 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.383159 3331 consensus_queue.cc:1035] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.385579 3318 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.386122 32210 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.386826 575 raft_consensus.cc:2932] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.390504 29747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 642312cdad114fce839d2b60717bcc72 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:58.392311 29756 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P 76c7a691de8a4e31b32799f43cb4b812 reported cstate change: config changed from index 8 to 9, VOTER 4a15e881772b41eab8a71fc027390db9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "76c7a691de8a4e31b32799f43cb4b812" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:58.397533 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 642312cdad114fce839d2b60717bcc72 with delete_type TABLET_DATA_TOMBSTONED (TS 4a15e881772b41eab8a71fc027390db9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53956
I20240501 14:01:58.398321 3337 tablet_replica.cc:330] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: stopping tablet replica
I20240501 14:01:58.398675 3337 raft_consensus.cc:2229] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:58.398931 3337 raft_consensus.cc:2258] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:58.399912 3337 ts_tablet_manager.cc:1875] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:58.408409 3337 ts_tablet_manager.cc:1888] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:58.408532 3337 log.cc:1196] T 642312cdad114fce839d2b60717bcc72 P 4a15e881772b41eab8a71fc027390db9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/642312cdad114fce839d2b60717bcc72
I20240501 14:01:58.409389 29746 catalog_manager.cc:4830] TS 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): tablet 642312cdad114fce839d2b60717bcc72 (table rebalance_test_table_1 [id=f7f8fa24b83940f9bd58d37ed26235fa]) successfully deleted
I20240501 14:01:58.432399 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56632
I20240501 14:01:58.432621 29901 raft_consensus.cc:595] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:58.568962 3062 raft_consensus.cc:1050] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:58.569953 3062 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.572726 32075 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.572945 30162 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.573247 575 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:58.573411 3358 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.573796 3062 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.574167 3360 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:58.577214 3360 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.577870 32075 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.578370 575 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.582149 30160 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.584290 29752 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } attrs { replace: true } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.590415 30290 consensus_queue.cc:237] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.592538 575 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.592861 32075 raft_consensus.cc:1261] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:58.593312 3062 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.593619 3360 consensus_queue.cc:1035] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:58.596621 3358 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.596879 575 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.597039 32075 raft_consensus.cc:2932] T f7d3626b1c8d458e9c6d4a28bc0564fa P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:58.599861 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7d3626b1c8d458e9c6d4a28bc0564fa with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:58.602427 29753 catalog_manager.cc:5484] T f7d3626b1c8d458e9c6d4a28bc0564fa P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 8 to 9, VOTER 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:58.606253 30141 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3626b1c8d458e9c6d4a28bc0564fa with delete_type TABLET_DATA_TOMBSTONED (TS 76c7a691de8a4e31b32799f43cb4b812 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240501 14:01:58.607465 3302 tablet_replica.cc:330] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: stopping tablet replica
I20240501 14:01:58.607697 3302 raft_consensus.cc:2229] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:58.608085 3302 raft_consensus.cc:2258] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:58.609074 3302 ts_tablet_manager.cc:1875] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:58.613876 3302 ts_tablet_manager.cc:1888] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:58.614006 3302 log.cc:1196] T f7d3626b1c8d458e9c6d4a28bc0564fa P 76c7a691de8a4e31b32799f43cb4b812: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:01:58.614814 29747 catalog_manager.cc:4830] TS 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): tablet f7d3626b1c8d458e9c6d4a28bc0564fa (table rebalance_test_table_8 [id=811dbb1a183347a98f97e2536c4f73f9]) successfully deleted
I20240501 14:01:58.619705 3258 raft_consensus.cc:979] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: : Instructing follower 4a15e881772b41eab8a71fc027390db9 to start an election
I20240501 14:01:58.619877 3258 raft_consensus.cc:1067] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Signalling peer 4a15e881772b41eab8a71fc027390db9 to start an election
I20240501 14:01:58.620600 30290 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536"
dest_uuid: "4a15e881772b41eab8a71fc027390db9"
from {username='slave'} at 127.22.5.193:32853
I20240501 14:01:58.620805 30290 raft_consensus.cc:482] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:58.620929 30290 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:58.622941 30290 raft_consensus.cc:504] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.623948 30290 leader_election.cc:288] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:01:58.624326 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:01:58.624415 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "744cef852ee9491db3928fa0092ef189"
I20240501 14:01:58.624588 29901 raft_consensus.cc:3032] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:58.624647 32210 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:58.624686 29901 raft_consensus.cc:729] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:01:58.625017 29901 consensus_queue.cc:260] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.625523 29901 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:58.627593 32210 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:58.628217 29901 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:58.629436 30228 leader_election.cc:302] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a15e881772b41eab8a71fc027390db9, 744cef852ee9491db3928fa0092ef189, 7869901e77234aeeb1462876b63eb28e; no voters:
I20240501 14:01:58.629499 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "4a15e881772b41eab8a71fc027390db9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "aa11715897034babb1009134d8914066"
I20240501 14:01:58.629794 575 raft_consensus.cc:3037] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:58.629817 3062 raft_consensus.cc:2781] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:58.630085 3062 raft_consensus.cc:686] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Becoming Leader. State: Replica: 4a15e881772b41eab8a71fc027390db9, State: Running, Role: LEADER
I20240501 14:01:58.630414 3062 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:58.632346 575 raft_consensus.cc:2454] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a15e881772b41eab8a71fc027390db9 in term 2.
I20240501 14:01:58.633908 29757 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: term changed from 1 to 2, leader changed from 7869901e77234aeeb1462876b63eb28e (127.22.5.193) to 4a15e881772b41eab8a71fc027390db9 (127.22.5.196). New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:58.988303 2943 tablet_copy_client.cc:523] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:58.991194 2943 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Bootstrap starting.
I20240501 14:01:59.024945 2943 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:59.025493 2943 tablet_bootstrap.cc:492] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Bootstrap complete.
I20240501 14:01:59.025799 2943 ts_tablet_manager.cc:1367] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.004s
I20240501 14:01:59.026527 2943 raft_consensus.cc:348] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:59.026726 2943 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa11715897034babb1009134d8914066, State: Initialized, Role: LEARNER
I20240501 14:01:59.026932 2943 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: true } }
I20240501 14:01:59.027441 2943 ts_tablet_manager.cc:1398] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:59.028324 29921 tablet_copy_service.cc:342] P 7869901e77234aeeb1462876b63eb28e: Request end of tablet copy session aa11715897034babb1009134d8914066-ef202920b8d64287a8a2cdb40c73a819 received from {username='slave'} at 127.22.5.201:54107
I20240501 14:01:59.028468 29921 tablet_copy_service.cc:433] P 7869901e77234aeeb1462876b63eb28e: ending tablet copy session aa11715897034babb1009134d8914066-ef202920b8d64287a8a2cdb40c73a819 on tablet ef202920b8d64287a8a2cdb40c73a819 with peer aa11715897034babb1009134d8914066
I20240501 14:01:59.084579 32210 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:59.085229 3062 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:59.091599 575 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:59.092187 3062 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:59.095800 29901 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:59.096647 3062 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:59.101505 30290 consensus_queue.cc:237] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:59.104022 575 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:59.104619 3358 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:59.106861 3062 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.107519 575 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.108395 32209 raft_consensus.cc:1261] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:59.110112 3256 consensus_queue.cc:1035] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:59.112159 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 582bddb40ece4d5b8dcd3e1c00a53536 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:59.112144 32210 raft_consensus.cc:2932] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.114012 29757 catalog_manager.cc:5484] T 582bddb40ece4d5b8dcd3e1c00a53536 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:59.119396 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet 582bddb40ece4d5b8dcd3e1c00a53536 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40362
I20240501 14:01:59.131752 3407 tablet_replica.cc:330] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:01:59.132011 3407 raft_consensus.cc:2229] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:59.132264 3407 raft_consensus.cc:2258] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:59.133435 3407 ts_tablet_manager.cc:1875] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:59.140194 3407 ts_tablet_manager.cc:1888] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:59.140419 3407 log.cc:1196] T 582bddb40ece4d5b8dcd3e1c00a53536 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:01:59.141894 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet 582bddb40ece4d5b8dcd3e1c00a53536 (table rebalance_test_table_2 [id=8ddd85a0fd70447c802f6bbfc5d4d44b]) successfully deleted
I20240501 14:01:59.482115 575 raft_consensus.cc:1203] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:59.898025 3442 raft_consensus.cc:1050] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: attempting to promote NON_VOTER aa11715897034babb1009134d8914066 to VOTER
I20240501 14:01:59.898762 3442 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:01:59.901330 32075 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:59.901340 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:59.901687 575 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 LEARNER]: Refusing update from remote peer 7869901e77234aeeb1462876b63eb28e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:59.902053 3444 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:59.902462 3412 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:59.902904 3442 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:59.913651 3412 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.914433 32075 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.915042 575 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.919965 30032 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:01:59.920037 29756 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e reported cstate change: config changed from index 7 to 8, aa11715897034babb1009134d8914066 (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7869901e77234aeeb1462876b63eb28e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:00.010556 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56736
I20240501 14:02:00.010743 29901 raft_consensus.cc:595] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:00.229597 29901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819"
dest_uuid: "7869901e77234aeeb1462876b63eb28e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56756
I20240501 14:02:00.229807 29901 raft_consensus.cc:595] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:00.301033 3444 raft_consensus.cc:979] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: : Instructing follower aa11715897034babb1009134d8914066 to start an election
I20240501 14:02:00.301270 3444 raft_consensus.cc:1067] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Signalling peer aa11715897034babb1009134d8914066 to start an election
I20240501 14:02:00.301935 575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819"
dest_uuid: "aa11715897034babb1009134d8914066"
from {username='slave'} at 127.22.5.193:50437
I20240501 14:02:00.302187 575 raft_consensus.cc:482] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:00.302309 575 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:00.304586 575 raft_consensus.cc:504] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:00.305531 575 leader_election.cc:288] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447)
I20240501 14:02:00.305977 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4"
I20240501 14:02:00.306224 30032 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:00.307533 29901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7869901e77234aeeb1462876b63eb28e"
I20240501 14:02:00.307773 29901 raft_consensus.cc:3032] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:00.307906 29901 raft_consensus.cc:729] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7869901e77234aeeb1462876b63eb28e, State: Running, Role: LEADER
I20240501 14:02:00.308182 29901 consensus_queue.cc:260] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:00.308636 29901 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:00.309944 30032 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:02:00.311720 29901 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:02:00.312197 479 leader_election.cc:302] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4, aa11715897034babb1009134d8914066; no voters:
I20240501 14:02:00.312531 3336 raft_consensus.cc:2781] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:00.313478 3336 raft_consensus.cc:686] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 2 LEADER]: Becoming Leader. State: Replica: aa11715897034babb1009134d8914066, State: Running, Role: LEADER
I20240501 14:02:00.313853 3336 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:00.317454 29757 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 reported cstate change: term changed from 1 to 2, leader changed from 7869901e77234aeeb1462876b63eb28e (127.22.5.193) to aa11715897034babb1009134d8914066 (127.22.5.201). New cstate: current_term: 2 leader_uuid: "aa11715897034babb1009134d8914066" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:00.319517 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef202920b8d64287a8a2cdb40c73a819" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "47b7351d2e9b4dbd9a09fd4133a97565"
I20240501 14:02:00.319813 32075 raft_consensus.cc:3037] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:00.322294 32075 raft_consensus.cc:2454] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:02:00.722941 32075 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:00.723634 3336 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:00.817687 29901 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:00.818370 3336 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:00.827591 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:00.841339 3336 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:01.302279 575 consensus_queue.cc:237] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:01.304267 32075 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:02:01.304337 30032 raft_consensus.cc:1261] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer aa11715897034babb1009134d8914066: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:02:01.304839 3487 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:02:01.305178 3491 consensus_queue.cc:1035] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:02:01.307530 3490 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:02:01.308120 30032 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:02:01.310834 29745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef202920b8d64287a8a2cdb40c73a819 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:01.311790 32075 raft_consensus.cc:2932] T ef202920b8d64287a8a2cdb40c73a819 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } }
I20240501 14:02:01.313056 29752 catalog_manager.cc:5484] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 reported cstate change: config changed from index 8 to 10, VOTER 7869901e77234aeeb1462876b63eb28e (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "aa11715897034babb1009134d8914066" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:01.318004 29880 tablet_service.cc:1514] Processing DeleteTablet for tablet ef202920b8d64287a8a2cdb40c73a819 with delete_type TABLET_DATA_TOMBSTONED (TS 7869901e77234aeeb1462876b63eb28e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40362
I20240501 14:02:01.318682 3515 tablet_replica.cc:330] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: stopping tablet replica
I20240501 14:02:01.318893 3515 raft_consensus.cc:2229] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:01.319083 3515 raft_consensus.cc:2258] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:01.319878 3515 ts_tablet_manager.cc:1875] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:01.324295 3515 ts_tablet_manager.cc:1888] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:02:01.325266 3515 log.cc:1196] T ef202920b8d64287a8a2cdb40c73a819 P 7869901e77234aeeb1462876b63eb28e: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef202920b8d64287a8a2cdb40c73a819
I20240501 14:02:01.326154 29744 catalog_manager.cc:4830] TS 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): tablet ef202920b8d64287a8a2cdb40c73a819 (table rebalance_test_table_3 [id=c4394c1c8e1e4d54965f358c09ed5cf0]) successfully deleted
I20240501 14:02:01.690400 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:01.695976 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:01.714645 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:01.715909 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:01.723363 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:01.743671 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:01.746837 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:01.753717 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:01.763621 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:01.892714 30290 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:02:01.895743 575 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:02:01.896020 30227 consensus_peers.cc:485] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2b6d21ea3184a6eb9db2b87589781b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:01.896399 3576 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:02:01.896955 30032 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:02:01.897733 3576 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:02:01.899622 3516 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:02:01.900167 575 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:02:01.901814 30032 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } } }
I20240501 14:02:01.905673 29758 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 7 to 8, NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) added. New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:02.346338 3598 ts_tablet_manager.cc:902] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Initiating tablet copy from peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425)
I20240501 14:02:02.348160 3598 tablet_copy_client.cc:308] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:39425
I20240501 14:02:02.348946 30312 tablet_copy_service.cc:140] P 4a15e881772b41eab8a71fc027390db9: Received BeginTabletCopySession request for tablet f2b6d21ea3184a6eb9db2b87589781b0 from peer 47b7351d2e9b4dbd9a09fd4133a97565 ({username='slave'} at 127.22.5.199:33017)
I20240501 14:02:02.349093 30312 tablet_copy_service.cc:161] P 4a15e881772b41eab8a71fc027390db9: Beginning new tablet copy session on tablet f2b6d21ea3184a6eb9db2b87589781b0 from peer 47b7351d2e9b4dbd9a09fd4133a97565 at {username='slave'} at 127.22.5.199:33017: session id = 47b7351d2e9b4dbd9a09fd4133a97565-f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:02:02.351189 30312 tablet_copy_source_session.cc:205] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:02.352614 3598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d21ea3184a6eb9db2b87589781b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:02.359441 3598 tablet_copy_client.cc:783] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:02.359761 3598 tablet_copy_client.cc:647] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:02.361449 3602 tablet_copy_client.cc:1002] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Injecting latency into file download: 1500
I20240501 14:02:03.861882 3598 tablet_copy_client.cc:523] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:03.864681 3598 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap starting.
I20240501 14:02:03.897459 3598 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:03.897887 3598 tablet_bootstrap.cc:492] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Bootstrap complete.
I20240501 14:02:03.898151 3598 ts_tablet_manager.cc:1367] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent bootstrapping tablet: real 0.034s user 0.035s sys 0.000s
I20240501 14:02:03.898766 3598 raft_consensus.cc:348] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:02:03.899003 3598 raft_consensus.cc:729] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 47b7351d2e9b4dbd9a09fd4133a97565, State: Initialized, Role: LEARNER
I20240501 14:02:03.899159 3598 consensus_queue.cc:260] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: NON_VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: true } }
I20240501 14:02:03.899641 3598 ts_tablet_manager.cc:1398] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:02:03.900564 30312 tablet_copy_service.cc:342] P 4a15e881772b41eab8a71fc027390db9: Request end of tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-f2b6d21ea3184a6eb9db2b87589781b0 received from {username='slave'} at 127.22.5.199:33017
I20240501 14:02:03.900718 30312 tablet_copy_service.cc:433] P 4a15e881772b41eab8a71fc027390db9: ending tablet copy session 47b7351d2e9b4dbd9a09fd4133a97565-f2b6d21ea3184a6eb9db2b87589781b0 on tablet f2b6d21ea3184a6eb9db2b87589781b0 with peer 47b7351d2e9b4dbd9a09fd4133a97565
I20240501 14:02:04.069921 32075 raft_consensus.cc:1203] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:02:04.514412 3663 raft_consensus.cc:1050] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9: attempting to promote NON_VOTER 47b7351d2e9b4dbd9a09fd4133a97565 to VOTER
I20240501 14:02:04.515357 3663 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:02:04.517993 30032 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:04.518064 575 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:04.518549 32075 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 LEARNER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:02:04.518626 3663 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:04.518988 3665 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:04.519377 3662 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:02:04.522157 3662 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.523056 575 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.523334 32075 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.526669 30032 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.528869 29753 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 8 to 9, 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:04.541177 30290 consensus_queue.cc:237] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:02:04.543341 575 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:02:04.543401 32075 raft_consensus.cc:1261] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Refusing update from remote peer 4a15e881772b41eab8a71fc027390db9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:02:04.543963 3665 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:02:04.544306 3663 consensus_queue.cc:1035] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:02:04.546862 3662 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.547497 575 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.547675 32075 raft_consensus.cc:2932] T f2b6d21ea3184a6eb9db2b87589781b0 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } }
I20240501 14:02:04.551366 29746 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2b6d21ea3184a6eb9db2b87589781b0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:04.551921 29757 catalog_manager.cc:5484] T f2b6d21ea3184a6eb9db2b87589781b0 P 4a15e881772b41eab8a71fc027390db9 reported cstate change: config changed from index 9 to 10, VOTER 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194) evicted. New cstate: current_term: 2 leader_uuid: "4a15e881772b41eab8a71fc027390db9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:04.556632 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b6d21ea3184a6eb9db2b87589781b0 with delete_type TABLET_DATA_TOMBSTONED (TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37382
I20240501 14:02:04.557410 3673 tablet_replica.cc:330] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: stopping tablet replica
I20240501 14:02:04.557693 3673 raft_consensus.cc:2229] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:04.557968 3673 raft_consensus.cc:2258] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:04.558893 3673 ts_tablet_manager.cc:1875] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:04.563983 3673 ts_tablet_manager.cc:1888] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:02:04.564095 3673 log.cc:1196] T f2b6d21ea3184a6eb9db2b87589781b0 P 7ac7d9aa0b5e49de9962e9067d2bc0e4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:02:04.564841 29747 catalog_manager.cc:4830] TS 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): tablet f2b6d21ea3184a6eb9db2b87589781b0 (table rebalance_test_table_0 [id=e94a3fc2e0c144058e4117e7616620ad]) successfully deleted
I20240501 14:02:04.797076 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:04.822402 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:04.822999 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:04.823959 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:04.828594 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:04.879626 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:04.900594 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:04.926043 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:04.969172 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:05.164958 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:05.182076 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:05.194517 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:05.199326 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:05.199810 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:05.219635 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:05.267916 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:05.315908 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:05.322039 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.258232 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:06.273880 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.292515 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.292948 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:06.339864 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:06.350025 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:06.371302 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:06.385514 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:06.441465 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.654774 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.682382 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:06.691443 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:06.707177 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:06.717994 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:06.718466 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:06.760558 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:06.780484 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:06.781816 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.032701 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.041447 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.047127 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:07.047580 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:07.067332 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:07.071838 29965 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:34471, user_credentials={real_user=slave}} blocked reactor thread for 56445.8us
I20240501 14:02:07.108213 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:07.135694 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:07.138114 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:07.198923 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.517356 32057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:02:07.517824 31926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.535663 30270 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:07.538910 32190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.589919 29880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:07.608661 30141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:07.631001 30010 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:07.641566 31793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:07.672942 547 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
15344f00f71e4ebc8023c865b2536f99 | 127.22.5.254:36941 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:44713 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3321c47e58ba49f09f4f036ac72fc133 | 127.22.5.198:36301 | HEALTHY | <none> | 0 | 0
47b7351d2e9b4dbd9a09fd4133a97565 | 127.22.5.199:40447 | HEALTHY | <none> | 2 | 0
4a15e881772b41eab8a71fc027390db9 | 127.22.5.196:39425 | HEALTHY | <none> | 7 | 0
744cef852ee9491db3928fa0092ef189 | 127.22.5.200:43981 | HEALTHY | <none> | 0 | 0
76c7a691de8a4e31b32799f43cb4b812 | 127.22.5.195:38823 | HEALTHY | <none> | 6 | 0
7869901e77234aeeb1462876b63eb28e | 127.22.5.193:35665 | HEALTHY | <none> | 7 | 0
7ac7d9aa0b5e49de9962e9067d2bc0e4 | 127.22.5.194:38885 | HEALTHY | <none> | 6 | 0
aa11715897034babb1009134d8914066 | 127.22.5.201:38269 | HEALTHY | <none> | 2 | 0
be55dad80e3a4ad79d21aaa3199b1d0b | 127.22.5.197:34471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.22.5.193 | experimental | 127.22.5.193:35665
local_ip_for_outbound_sockets | 127.22.5.194 | experimental | 127.22.5.194:38885
local_ip_for_outbound_sockets | 127.22.5.195 | experimental | 127.22.5.195:38823
local_ip_for_outbound_sockets | 127.22.5.196 | experimental | 127.22.5.196:39425
local_ip_for_outbound_sockets | 127.22.5.197 | experimental | 127.22.5.197:34471
local_ip_for_outbound_sockets | 127.22.5.198 | experimental | 127.22.5.198:36301
local_ip_for_outbound_sockets | 127.22.5.199 | experimental | 127.22.5.199:40447
local_ip_for_outbound_sockets | 127.22.5.200 | experimental | 127.22.5.200:43981
local_ip_for_outbound_sockets | 127.22.5.201 | experimental | 127.22.5.201:38269
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.5.193:35665
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.5.194:38885
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.5.195:38823
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.5.196:39425
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.5.197:34471
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.5.198:36301
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.5.199:40447
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.5.200:43981
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.22.5.201:38269
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:44713 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:02:07.749181 22551 log_verifier.cc:126] Checking tablet 0929994d486d4610b9cc76abd39f043a
I20240501 14:02:07.791029 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 0929994d486d4610b9cc76abd39f043a
I20240501 14:02:07.791167 22551 log_verifier.cc:126] Checking tablet 0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:02:07.832340 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 0ea9aaebd437424eb27c04bc2f6ffa7b
I20240501 14:02:07.832470 22551 log_verifier.cc:126] Checking tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a
I20240501 14:02:07.872959 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1ccf3fc5fc7f44a3bc53dad7902c2d3a
I20240501 14:02:07.873075 22551 log_verifier.cc:126] Checking tablet 212e139f1e7c4674987c2309efe3e495
I20240501 14:02:07.949569 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 212e139f1e7c4674987c2309efe3e495
I20240501 14:02:07.949674 22551 log_verifier.cc:126] Checking tablet 2b1ada0fe1ce431a823f2814f116db28
I20240501 14:02:07.988837 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 2b1ada0fe1ce431a823f2814f116db28
I20240501 14:02:07.988974 22551 log_verifier.cc:126] Checking tablet 31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:02:08.027917 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 31303ad8c9124217a45e26f8da5cc0a1
I20240501 14:02:08.028050 22551 log_verifier.cc:126] Checking tablet 368edc181c894686a213fc542f328192
I20240501 14:02:08.067332 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 368edc181c894686a213fc542f328192
I20240501 14:02:08.067484 22551 log_verifier.cc:126] Checking tablet 427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:02:08.106444 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 427af2a000f34ab6b9e3c9c7d7be1b32
I20240501 14:02:08.106577 22551 log_verifier.cc:126] Checking tablet 48b5a3abfd8348b9ae47c8867642044e
I20240501 14:02:08.144899 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 48b5a3abfd8348b9ae47c8867642044e
I20240501 14:02:08.145018 22551 log_verifier.cc:126] Checking tablet 4cc0201075e1466a9661b155c180d95f
I20240501 14:02:08.217756 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4cc0201075e1466a9661b155c180d95f
I20240501 14:02:08.217859 22551 log_verifier.cc:126] Checking tablet 56f632889b1e400392fe710bf9f6f3ec
I20240501 14:02:08.255736 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 56f632889b1e400392fe710bf9f6f3ec
I20240501 14:02:08.255860 22551 log_verifier.cc:126] Checking tablet 582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:02:08.294898 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 582bddb40ece4d5b8dcd3e1c00a53536
I20240501 14:02:08.295096 22551 log_verifier.cc:126] Checking tablet 594dcb0a05054f2abaa11e5d4537b724
I20240501 14:02:08.337412 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 594dcb0a05054f2abaa11e5d4537b724
I20240501 14:02:08.337536 22551 log_verifier.cc:126] Checking tablet 5c35ccca22fb45b78ee66991027c0261
I20240501 14:02:08.385165 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5c35ccca22fb45b78ee66991027c0261
I20240501 14:02:08.385303 22551 log_verifier.cc:126] Checking tablet 6241869f8e7f4bfaa65508fd61f405a6
I20240501 14:02:08.425225 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6241869f8e7f4bfaa65508fd61f405a6
I20240501 14:02:08.425426 22551 log_verifier.cc:126] Checking tablet 64012de915564c7887e44d432b170fdc
I20240501 14:02:08.500666 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 64012de915564c7887e44d432b170fdc
I20240501 14:02:08.500787 22551 log_verifier.cc:126] Checking tablet 642312cdad114fce839d2b60717bcc72
I20240501 14:02:08.538746 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 642312cdad114fce839d2b60717bcc72
I20240501 14:02:08.538857 22551 log_verifier.cc:126] Checking tablet 9735c5d9f784459ebafd8f2ff4f69763
I20240501 14:02:08.577816 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9735c5d9f784459ebafd8f2ff4f69763
I20240501 14:02:08.577929 22551 log_verifier.cc:126] Checking tablet 98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:02:08.616015 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 98d94ed11f0f4a9d9f519a24c3320ce4
I20240501 14:02:08.616128 22551 log_verifier.cc:126] Checking tablet 9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:02:08.658102 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9b56df9dcda34e6a8d27a8a6b24742b2
I20240501 14:02:08.658241 22551 log_verifier.cc:126] Checking tablet a733857ad51d452a9054cc5f533edc8e
I20240501 14:02:08.698450 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a733857ad51d452a9054cc5f533edc8e
I20240501 14:02:08.698549 22551 log_verifier.cc:126] Checking tablet c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:02:08.769766 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet c00649352f9d41c3a12a5e767b8b2aa6
I20240501 14:02:08.769949 22551 log_verifier.cc:126] Checking tablet c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:02:08.809486 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c40a79b118424a5699bddd1c4ac4bb0e
I20240501 14:02:08.809588 22551 log_verifier.cc:126] Checking tablet c953ab40905a4d23b9b9c970d25168b9
I20240501 14:02:08.851830 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c953ab40905a4d23b9b9c970d25168b9
I20240501 14:02:08.851936 22551 log_verifier.cc:126] Checking tablet d1bde99cadd94efc826b4074b2674104
I20240501 14:02:08.890277 22551 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d1bde99cadd94efc826b4074b2674104
I20240501 14:02:08.890384 22551 log_verifier.cc:126] Checking tablet edf4bcf02c7e4352bd998e139802db62
I20240501 14:02:08.929729 22551 log_verifier.cc:177] Verified matching terms for 7 ops in tablet edf4bcf02c7e4352bd998e139802db62
I20240501 14:02:08.929831 22551 log_verifier.cc:126] Checking tablet ef202920b8d64287a8a2cdb40c73a819
I20240501 14:02:08.969271 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet ef202920b8d64287a8a2cdb40c73a819
I20240501 14:02:08.969417 22551 log_verifier.cc:126] Checking tablet f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:02:09.045074 22551 log_verifier.cc:177] Verified matching terms for 10 ops in tablet f2b6d21ea3184a6eb9db2b87589781b0
I20240501 14:02:09.045220 22551 log_verifier.cc:126] Checking tablet f4988f0de3214860a65007b8e31e88a5
I20240501 14:02:09.083031 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f4988f0de3214860a65007b8e31e88a5
I20240501 14:02:09.083138 22551 log_verifier.cc:126] Checking tablet f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:02:09.121760 22551 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f7d3626b1c8d458e9c6d4a28bc0564fa
I20240501 14:02:09.122959 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 29819
I20240501 14:02:09.134479 29940 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:09.384658 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 29819
W20240501 14:02:09.432152 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.5.193:35665 (error 108)
W20240501 14:02:09.432439 31879 connection.cc:535] server connection from 127.22.5.193:58507 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:09.432956 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 29950
W20240501 14:02:09.435501 30095 consensus_peers.cc:485] T 48b5a3abfd8348b9ae47c8867642044e P 76c7a691de8a4e31b32799f43cb4b812 -> Peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Couldn't send request to peer 7869901e77234aeeb1462876b63eb28e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.440182 30072 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:09.664573 479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
W20240501 14:02:09.666049 479 consensus_peers.cc:485] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 -> Peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Couldn't send request to peer 7869901e77234aeeb1462876b63eb28e. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.674875 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 29950
W20240501 14:02:09.716879 479 connection.cc:535] server connection from 127.22.5.194:38879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:09.716912 30098 connection.cc:535] client connection to 127.22.5.194:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:09.717357 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30080
W20240501 14:02:09.717499 29747 connection.cc:535] server connection from 127.22.5.194:59949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:09.718191 30098 consensus_peers.cc:485] T edf4bcf02c7e4352bd998e139802db62 P 76c7a691de8a4e31b32799f43cb4b812 -> Peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Couldn't send request to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.723866 30201 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:09.742518 482 consensus_peers.cc:485] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 -> Peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Couldn't send request to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.935086 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30080
W20240501 14:02:09.949651 32013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111)
W20240501 14:02:09.951222 32013 consensus_peers.cc:485] T 31303ad8c9124217a45e26f8da5cc0a1 P 47b7351d2e9b4dbd9a09fd4133a97565 -> Peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Couldn't send request to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:09.974534 31881 connection.cc:535] server connection from 127.22.5.195:39375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:09.974845 30227 connection.cc:535] server connection from 127.22.5.195:42331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:09.974932 32143 connection.cc:535] server connection from 127.22.5.195:36327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:09.975371 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 30210
W20240501 14:02:09.975301 30228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.5.195:38823 (error 108)
W20240501 14:02:09.974686 32013 connection.cc:535] server connection from 127.22.5.195:38601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:09.978158 30228 consensus_peers.cc:485] T 64012de915564c7887e44d432b170fdc P 4a15e881772b41eab8a71fc027390db9 -> Peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Couldn't send request to peer 76c7a691de8a4e31b32799f43cb4b812. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.195:38823: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:09.982585 30330 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:10.151710 3976 raft_consensus.cc:482] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.152001 3976 raft_consensus.cc:504] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.152983 3976 leader_election.cc:288] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
W20240501 14:02:10.155506 479 leader_election.cc:334] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.160985 31941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "3321c47e58ba49f09f4f036ac72fc133" is_pre_election: true
I20240501 14:02:10.161772 482 leader_election.cc:302] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa11715897034babb1009134d8914066; no voters: 3321c47e58ba49f09f4f036ac72fc133, 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.162073 3976 raft_consensus.cc:2726] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.202639 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 30210
I20240501 14:02:10.221047 3981 raft_consensus.cc:482] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.221447 3981 raft_consensus.cc:504] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.223076 3981 leader_election.cc:288] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
W20240501 14:02:10.224850 31879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
W20240501 14:02:10.227409 31879 leader_election.cc:334] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.230508 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "3321c47e58ba49f09f4f036ac72fc133" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "aa11715897034babb1009134d8914066" is_pre_election: true
I20240501 14:02:10.230810 575 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3321c47e58ba49f09f4f036ac72fc133 in term 1.
I20240501 14:02:10.231314 31880 leader_election.cc:302] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3321c47e58ba49f09f4f036ac72fc133, aa11715897034babb1009134d8914066; no voters: 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.231775 3981 raft_consensus.cc:2781] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:10.231920 3981 raft_consensus.cc:482] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.232026 3981 raft_consensus.cc:3037] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:10.234268 3981 raft_consensus.cc:504] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.234920 3981 leader_election.cc:288] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:02:10.235517 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368edc181c894686a213fc542f328192" candidate_uuid: "3321c47e58ba49f09f4f036ac72fc133" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "aa11715897034babb1009134d8914066"
I20240501 14:02:10.235781 575 raft_consensus.cc:3037] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:02:10.236339 31879 leader_election.cc:334] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.239279 575 raft_consensus.cc:2454] T 368edc181c894686a213fc542f328192 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3321c47e58ba49f09f4f036ac72fc133 in term 2.
I20240501 14:02:10.239826 31880 leader_election.cc:302] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3321c47e58ba49f09f4f036ac72fc133, aa11715897034babb1009134d8914066; no voters: 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.240187 3981 raft_consensus.cc:2781] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:10.240629 3981 raft_consensus.cc:686] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [term 2 LEADER]: Becoming Leader. State: Replica: 3321c47e58ba49f09f4f036ac72fc133, State: Running, Role: LEADER
I20240501 14:02:10.241202 3981 consensus_queue.cc:237] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.245513 29758 catalog_manager.cc:5484] T 368edc181c894686a213fc542f328192 P 3321c47e58ba49f09f4f036ac72fc133 reported cstate change: term changed from 1 to 2, leader changed from 7869901e77234aeeb1462876b63eb28e (127.22.5.193) to 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198). New cstate: current_term: 2 leader_uuid: "3321c47e58ba49f09f4f036ac72fc133" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:02:10.246755 32012 connection.cc:535] client connection to 127.22.5.196:39425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:10.248858 3969 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:10.247467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.5.196:39425 (local address 127.22.5.199:48293)
0501 14:02:10.247777 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:10.247863 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.5.196:39425: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":171}
W20240501 14:02:10.249091 32012 consensus_peers.cc:485] T 2b1ada0fe1ce431a823f2814f116db28 P 47b7351d2e9b4dbd9a09fd4133a97565 -> Peer 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425): Couldn't send request to peer 4a15e881772b41eab8a71fc027390db9. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.196:39425: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:10.249604 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 31731
I20240501 14:02:10.256759 31852 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:10.326189 3981 raft_consensus.cc:482] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.326464 3981 raft_consensus.cc:504] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.327153 3981 leader_election.cc:288] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:02:10.327862 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea9aaebd437424eb27c04bc2f6ffa7b" candidate_uuid: "3321c47e58ba49f09f4f036ac72fc133" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "aa11715897034babb1009134d8914066" is_pre_election: true
W20240501 14:02:10.328732 31879 leader_election.cc:334] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.328852 31879 leader_election.cc:302] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3321c47e58ba49f09f4f036ac72fc133; no voters: 7869901e77234aeeb1462876b63eb28e, aa11715897034babb1009134d8914066
I20240501 14:02:10.329156 3981 raft_consensus.cc:2726] T 0ea9aaebd437424eb27c04bc2f6ffa7b P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.370541 3491 raft_consensus.cc:482] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.371049 3491 raft_consensus.cc:504] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.372159 3491 leader_election.cc:288] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
W20240501 14:02:10.374333 479 leader_election.cc:334] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.397321 3981 raft_consensus.cc:482] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.397586 3981 raft_consensus.cc:504] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:02:10.398427 3981 leader_election.cc:288] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
W20240501 14:02:10.400820 31879 leader_election.cc:334] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.406049 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c35ccca22fb45b78ee66991027c0261" candidate_uuid: "3321c47e58ba49f09f4f036ac72fc133" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "744cef852ee9491db3928fa0092ef189" is_pre_election: true
I20240501 14:02:10.406821 31882 leader_election.cc:302] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3321c47e58ba49f09f4f036ac72fc133; no voters: 744cef852ee9491db3928fa0092ef189, 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.407203 3981 raft_consensus.cc:2726] T 5c35ccca22fb45b78ee66991027c0261 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.422374 31813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212e139f1e7c4674987c2309efe3e495" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" is_pre_election: true
I20240501 14:02:10.422500 3997 raft_consensus.cc:482] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.422752 3997 raft_consensus.cc:504] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.423667 31813 raft_consensus.cc:2454] T 212e139f1e7c4674987c2309efe3e495 P be55dad80e3a4ad79d21aaa3199b1d0b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 1.
I20240501 14:02:10.425098 479 leader_election.cc:302] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa11715897034babb1009134d8914066, be55dad80e3a4ad79d21aaa3199b1d0b; no voters: 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.425881 3491 raft_consensus.cc:2781] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:10.425993 3491 raft_consensus.cc:482] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.426108 3491 raft_consensus.cc:3037] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:10.429260 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 31731
I20240501 14:02:10.431123 3981 raft_consensus.cc:482] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ac7d9aa0b5e49de9962e9067d2bc0e4)
I20240501 14:02:10.431375 3981 raft_consensus.cc:504] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:02:10.432257 3981 leader_election.cc:288] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:02:10.432323 3491 raft_consensus.cc:504] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "be55dad80e3a4ad79d21aaa3199b1d0b" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.433537 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4988f0de3214860a65007b8e31e88a5" candidate_uuid: "3321c47e58ba49f09f4f036ac72fc133" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "744cef852ee9491db3928fa0092ef189" is_pre_election: true
W20240501 14:02:10.434147 31882 leader_election.cc:334] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111)
I20240501 14:02:10.434180 3491 leader_election.cc:288] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Requested vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471)
I20240501 14:02:10.434430 31882 leader_election.cc:302] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3321c47e58ba49f09f4f036ac72fc133; no voters: 744cef852ee9491db3928fa0092ef189, 7ac7d9aa0b5e49de9962e9067d2bc0e4
W20240501 14:02:10.434700 479 leader_election.cc:334] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.434811 3981 raft_consensus.cc:2726] T f4988f0de3214860a65007b8e31e88a5 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:10.458520 479 connection.cc:535] client connection to 127.22.5.197:34471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:10.458895 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 31862
W20240501 14:02:10.459941 479 leader_election.cc:334] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Network error: Client connection negotiation failed: client connection to 127.22.5.197:34471: connect: Connection refused (error 111)
I20240501 14:02:10.460104 479 leader_election.cc:302] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa11715897034babb1009134d8914066; no voters: 7869901e77234aeeb1462876b63eb28e, be55dad80e3a4ad79d21aaa3199b1d0b
I20240501 14:02:10.460469 3491 raft_consensus.cc:2726] T 212e139f1e7c4674987c2309efe3e495 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 14:02:10.460589 479 consensus_peers.cc:485] T c00649352f9d41c3a12a5e767b8b2aa6 P aa11715897034babb1009134d8914066 -> Peer be55dad80e3a4ad79d21aaa3199b1d0b (127.22.5.197:34471): Couldn't send request to peer be55dad80e3a4ad79d21aaa3199b1d0b. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.197:34471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:10.464895 31984 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:10.549448 4000 raft_consensus.cc:482] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.549713 4000 raft_consensus.cc:504] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:02:10.551491 4000 leader_election.cc:288] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
W20240501 14:02:10.552716 32143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
W20240501 14:02:10.555044 32143 leader_election.cc:334] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.623126 3981 raft_consensus.cc:482] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.623764 3981 raft_consensus.cc:504] T c953ab40905a4d23b9b9c970d25168b9 P 3321c47e58ba49f09f4f036ac72fc133 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
I20240501 14:02:10.624270 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 31862
I20240501 14:02:10.636734 3995 raft_consensus.cc:482] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.636993 3995 raft_consensus.cc:504] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } }
I20240501 14:02:10.638198 3995 leader_election.cc:288] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
W20240501 14:02:10.639802 32010 leader_election.cc:334] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
W20240501 14:02:10.640126 32013 leader_election.cc:334] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111)
I20240501 14:02:10.640290 32013 leader_election.cc:302] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b7351d2e9b4dbd9a09fd4133a97565; no voters: 7869901e77234aeeb1462876b63eb28e, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:02:10.640604 3995 raft_consensus.cc:2726] T 1ccf3fc5fc7f44a3bc53dad7902c2d3a P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.647177 4000 raft_consensus.cc:482] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ac7d9aa0b5e49de9962e9067d2bc0e4)
I20240501 14:02:10.647421 4000 raft_consensus.cc:504] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:02:10.648439 4000 leader_election.cc:288] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885)
W20240501 14:02:10.650460 32146 leader_election.cc:334] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Network error: Client connection negotiation failed: client connection to 127.22.5.195:38823: connect: Connection refused (error 111)
W20240501 14:02:10.650651 32146 leader_election.cc:334] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885): Network error: Client connection negotiation failed: client connection to 127.22.5.194:38885: connect: Connection refused (error 111)
I20240501 14:02:10.650770 32146 leader_election.cc:302] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744cef852ee9491db3928fa0092ef189; no voters: 76c7a691de8a4e31b32799f43cb4b812, 7ac7d9aa0b5e49de9962e9067d2bc0e4
I20240501 14:02:10.651206 4000 raft_consensus.cc:2726] T 9735c5d9f784459ebafd8f2ff4f69763 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.654778 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 31994
W20240501 14:02:10.655262 4002 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:10.551770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.5.198:36301 (local address 127.22.5.200:58055)
0501 14:02:10.553271 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:10.553297 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:10.553479 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:10.624333 (+ 70854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 14:02:10.624340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 14:02:10.624380 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 14:02:10.625155 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 14:02:10.625171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 14:02:10.654269 (+ 29098us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.5.198:36301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":591,"threads_started":1}
W20240501 14:02:10.656567 32146 leader_election.cc:334] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Network error: Client connection negotiation failed: client connection to 127.22.5.198:36301: connect: Connection refused (error 111)
I20240501 14:02:10.656728 32146 leader_election.cc:302] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744cef852ee9491db3928fa0092ef189; no voters: 3321c47e58ba49f09f4f036ac72fc133, 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.657020 4000 raft_consensus.cc:2726] T 5c35ccca22fb45b78ee66991027c0261 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.661216 32116 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:10.831291 4000 raft_consensus.cc:482] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76c7a691de8a4e31b32799f43cb4b812)
I20240501 14:02:10.831538 4000 raft_consensus.cc:504] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.832414 4000 leader_election.cc:288] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:02:10.834731 3995 raft_consensus.cc:482] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ac7d9aa0b5e49de9962e9067d2bc0e4)
I20240501 14:02:10.834947 3995 raft_consensus.cc:504] T 427af2a000f34ab6b9e3c9c7d7be1b32 P 47b7351d2e9b4dbd9a09fd4133a97565 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "47b7351d2e9b4dbd9a09fd4133a97565" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 40447 } attrs { promote: false } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } }
W20240501 14:02:10.835515 32146 leader_election.cc:334] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Network error: Client connection negotiation failed: client connection to 127.22.5.195:38823: connect: Connection refused (error 111)
I20240501 14:02:10.835690 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 31994
I20240501 14:02:10.842170 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "744cef852ee9491db3928fa0092ef189" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "aa11715897034babb1009134d8914066" is_pre_election: true
I20240501 14:02:10.842943 32144 leader_election.cc:302] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744cef852ee9491db3928fa0092ef189; no voters: 76c7a691de8a4e31b32799f43cb4b812, aa11715897034babb1009134d8914066
I20240501 14:02:10.843292 4000 raft_consensus.cc:2726] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:10.843325 3491 raft_consensus.cc:482] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76c7a691de8a4e31b32799f43cb4b812)
I20240501 14:02:10.843495 3491 raft_consensus.cc:504] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.844224 3491 leader_election.cc:288] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
W20240501 14:02:10.846138 482 leader_election.cc:334] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Network error: Client connection negotiation failed: client connection to 127.22.5.195:38823: connect: Connection refused (error 111)
I20240501 14:02:10.850489 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "744cef852ee9491db3928fa0092ef189" is_pre_election: true
I20240501 14:02:10.850744 32210 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 1.
I20240501 14:02:10.851147 482 leader_election.cc:302] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 744cef852ee9491db3928fa0092ef189, aa11715897034babb1009134d8914066; no voters: 76c7a691de8a4e31b32799f43cb4b812
I20240501 14:02:10.851457 3491 raft_consensus.cc:2781] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:10.851580 3491 raft_consensus.cc:482] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76c7a691de8a4e31b32799f43cb4b812)
I20240501 14:02:10.851676 3491 raft_consensus.cc:3037] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:10.854053 3491 raft_consensus.cc:504] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.854729 3491 leader_election.cc:288] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Requested vote from peers 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823), 744cef852ee9491db3928fa0092ef189 (127.22.5.200:43981)
I20240501 14:02:10.855633 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642312cdad114fce839d2b60717bcc72" candidate_uuid: "aa11715897034babb1009134d8914066" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "744cef852ee9491db3928fa0092ef189"
I20240501 14:02:10.855861 32210 raft_consensus.cc:3037] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:10.857954 32210 raft_consensus.cc:2454] T 642312cdad114fce839d2b60717bcc72 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa11715897034babb1009134d8914066 in term 2.
I20240501 14:02:10.859362 482 leader_election.cc:302] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744cef852ee9491db3928fa0092ef189, aa11715897034babb1009134d8914066; no voters:
I20240501 14:02:10.859683 3491 raft_consensus.cc:2781] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:10.859835 3491 raft_consensus.cc:686] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [term 2 LEADER]: Becoming Leader. State: Replica: aa11715897034babb1009134d8914066, State: Running, Role: LEADER
W20240501 14:02:10.860177 482 leader_election.cc:334] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195:38823): Network error: Client connection negotiation failed: client connection to 127.22.5.195:38823: connect: Connection refused (error 111)
I20240501 14:02:10.860178 3491 consensus_queue.cc:237] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.863672 29758 catalog_manager.cc:5484] T 642312cdad114fce839d2b60717bcc72 P aa11715897034babb1009134d8914066 reported cstate change: term changed from 1 to 2, leader changed from 76c7a691de8a4e31b32799f43cb4b812 (127.22.5.195) to aa11715897034babb1009134d8914066 (127.22.5.201). New cstate: current_term: 2 leader_uuid: "aa11715897034babb1009134d8914066" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "76c7a691de8a4e31b32799f43cb4b812" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 38823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:10.870601 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 32126
I20240501 14:02:10.877511 32250 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:10.883577 4010 raft_consensus.cc:482] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7869901e77234aeeb1462876b63eb28e)
I20240501 14:02:10.883867 4010 raft_consensus.cc:504] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7869901e77234aeeb1462876b63eb28e" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 35665 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:10.884748 4010 leader_election.cc:288] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665), 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
W20240501 14:02:10.886317 482 leader_election.cc:334] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301): Network error: Client connection negotiation failed: client connection to 127.22.5.198:36301: connect: Connection refused (error 111)
W20240501 14:02:10.886521 479 leader_election.cc:334] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7869901e77234aeeb1462876b63eb28e (127.22.5.193:35665): Network error: Client connection negotiation failed: client connection to 127.22.5.193:35665: connect: Connection refused (error 111)
I20240501 14:02:10.886668 479 leader_election.cc:302] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa11715897034babb1009134d8914066; no voters: 3321c47e58ba49f09f4f036ac72fc133, 7869901e77234aeeb1462876b63eb28e
I20240501 14:02:10.886986 4010 raft_consensus.cc:2726] T 0ea9aaebd437424eb27c04bc2f6ffa7b P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:11.031154 481 consensus_peers.cc:485] T ef202920b8d64287a8a2cdb40c73a819 P aa11715897034babb1009134d8914066 -> Peer 47b7351d2e9b4dbd9a09fd4133a97565 (127.22.5.199:40447): Couldn't send request to peer 47b7351d2e9b4dbd9a09fd4133a97565. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.199:40447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:11.031206 4000 raft_consensus.cc:482] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a15e881772b41eab8a71fc027390db9)
I20240501 14:02:11.031433 4000 raft_consensus.cc:504] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:11.031821 4017 raft_consensus.cc:482] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ac7d9aa0b5e49de9962e9067d2bc0e4)
I20240501 14:02:11.032001 4017 raft_consensus.cc:504] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } }
I20240501 14:02:11.032339 4000 leader_election.cc:288] T 582bddb40ece4d5b8dcd3e1c00a53536 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a15e881772b41eab8a71fc027390db9 (127.22.5.196:39425), aa11715897034babb1009134d8914066 (127.22.5.201:38269)
I20240501 14:02:11.032918 4017 leader_election.cc:288] T f4988f0de3214860a65007b8e31e88a5 P 744cef852ee9491db3928fa0092ef189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ac7d9aa0b5e49de9962e9067d2bc0e4 (127.22.5.194:38885), 3321c47e58ba49f09f4f036ac72fc133 (127.22.5.198:36301)
I20240501 14:02:11.032907 575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582bddb40ece4d5b8dcd3e1c00a53536" candidate_uuid: "744cef852ee9491db3928fa0092ef189" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "aa11715897034babb1009134d8914066" is_pre_election: true
I20240501 14:02:11.033254 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 32126
W20240501 14:02:11.059970 482 connection.cc:535] server connection from 127.22.5.200:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:11.060189 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 416
I20240501 14:02:11.066098 674 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:11.203819 3491 raft_consensus.cc:482] T f7d3626b1c8d458e9c6d4a28bc0564fa P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a15e881772b41eab8a71fc027390db9)
I20240501 14:02:11.204231 4016 raft_consensus.cc:482] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a15e881772b41eab8a71fc027390db9)
I20240501 14:02:11.204519 4010 raft_consensus.cc:482] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ac7d9aa0b5e49de9962e9067d2bc0e4)
I20240501 14:02:11.204422 4016 raft_consensus.cc:504] T 582bddb40ece4d5b8dcd3e1c00a53536 P aa11715897034babb1009134d8914066 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a15e881772b41eab8a71fc027390db9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 39425 } } peers { permanent_uuid: "744cef852ee9491db3928fa0092ef189" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:11.204898 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 416
I20240501 14:02:11.204764 4010 raft_consensus.cc:504] T 0929994d486d4610b9cc76abd39f043a P aa11715897034babb1009134d8914066 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ac7d9aa0b5e49de9962e9067d2bc0e4" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 38885 } } peers { permanent_uuid: "3321c47e58ba49f09f4f036ac72fc133" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 36301 } attrs { promote: false } } peers { permanent_uuid: "aa11715897034babb1009134d8914066" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 38269 } attrs { promote: false } }
I20240501 14:02:11.229669 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 29729
I20240501 14:02:11.230729 29790 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:11.436717 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 29729
2024-05-01T14:02:11Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (55514 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (55514 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240501 14:02:11.611812 22551 test_util.cc:268] Using random seed: -287787905
I20240501 14:02:11.613767 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:02:11.613857 22551 ts_itest-base.cc:113] --------------
I20240501 14:02:11.613903 22551 ts_itest-base.cc:114] 4 tablet servers
I20240501 14:02:11.613968 22551 ts_itest-base.cc:115] 1 replicas per TS
I20240501 14:02:11.614022 22551 ts_itest-base.cc:116] --------------
2024-05-01T14:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:11Z Disabled control of system clock
I20240501 14:02:11.675812 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:42339
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:42339
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 14:02:12.045704 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:12.046061 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:12.046159 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:12.054325 4027 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:12.054481 4027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:12.054530 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:12.054587 4027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:12.054637 4027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:12.064054 4027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:42339
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:12.065526 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:12.067625 4027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:12.079933 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:12.080448 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:12.081056 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:12.081574 4027 server_base.cc:1034] running on GCE node
I20240501 14:02:12.082383 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:12.083887 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:12.085091 4027 hybrid_clock.cc:648] HybridClock initialized: now 1714572132085052 us; error 41 us; skew 500 ppm
I20240501 14:02:12.085465 4027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:12.088097 4027 webserver.cc:460] Webserver started at http://127.22.5.254:43781/ using document root <none> and password file <none>
I20240501 14:02:12.088912 4027 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:12.089040 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:12.089426 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:12.092188 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8881fbbc87b3422095055ffd66449d63"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.093006 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8881fbbc87b3422095055ffd66449d63"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.098804 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:12.102478 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:12.103904 4027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:12.104081 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8881fbbc87b3422095055ffd66449d63"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.104300 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:12.133757 4027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:12.134677 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:12.134963 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:12.153677 4027 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:42339
I20240501 14:02:12.153697 4092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:42339 every 8 connection(s)
I20240501 14:02:12.155176 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:02:12.158990 4093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.162763 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4027
I20240501 14:02:12.162959 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:02:12.170262 4093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63: Bootstrap starting.
I20240501 14:02:12.173928 4093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.175318 4093 log.cc:824] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:12.178531 4093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63: No bootstrap required, opened a new log
I20240501 14:02:12.184976 4093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } }
I20240501 14:02:12.185415 4093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.185518 4093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8881fbbc87b3422095055ffd66449d63, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.186123 4093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } }
I20240501 14:02:12.186337 4093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:12.186489 4093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:12.186669 4093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.188835 4093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } }
I20240501 14:02:12.189360 4093 leader_election.cc:302] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8881fbbc87b3422095055ffd66449d63; no voters:
I20240501 14:02:12.189939 4093 leader_election.cc:288] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:12.190073 4098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:12.190721 4098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [term 1 LEADER]: Becoming Leader. State: Replica: 8881fbbc87b3422095055ffd66449d63, State: Running, Role: LEADER
I20240501 14:02:12.191298 4098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } }
I20240501 14:02:12.191967 4093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:12.194041 4100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8881fbbc87b3422095055ffd66449d63. Latest consensus state: current_term: 1 leader_uuid: "8881fbbc87b3422095055ffd66449d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } } }
I20240501 14:02:12.194444 4100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:12.194296 4099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8881fbbc87b3422095055ffd66449d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881fbbc87b3422095055ffd66449d63" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 42339 } } }
I20240501 14:02:12.195312 4099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:12.196841 4105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:12.202126 4105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:12.210769 4105 catalog_manager.cc:1347] Generated new cluster ID: da33247891c547b88f9134a4f37db830
I20240501 14:02:12.210899 4105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:12.234629 4105 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:12.235971 4105 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:12.243769 4105 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8881fbbc87b3422095055ffd66449d63: Generated new TSK 0
I20240501 14:02:12.244621 4105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:12.257390 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:02:12.639933 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:12.640261 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:12.640401 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:12.648164 4117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:12.648312 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:12.648494 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 14:02:12.657938 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:12.659390 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:12.661314 4117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:12.674232 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:12.674811 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:12.675163 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:12.675812 4117 server_base.cc:1034] running on GCE node
I20240501 14:02:12.676492 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:12.677698 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:12.678936 4117 hybrid_clock.cc:648] HybridClock initialized: now 1714572132678876 us; error 49 us; skew 500 ppm
I20240501 14:02:12.679325 4117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:12.681936 4117 webserver.cc:460] Webserver started at http://127.22.5.193:40351/ using document root <none> and password file <none>
I20240501 14:02:12.682791 4117 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:12.682920 4117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:12.683260 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:12.686199 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eb9930dcb1e94a7ea471bd95baa3c33d"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.687076 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eb9930dcb1e94a7ea471bd95baa3c33d"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.693420 4117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240501 14:02:12.697503 4131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:12.699049 4117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:12.699265 4117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eb9930dcb1e94a7ea471bd95baa3c33d"
format_stamp: "Formatted at 2024-05-01 14:02:12 on dist-test-slave-5jzj"
I20240501 14:02:12.699487 4117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:12.736032 4117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:12.737025 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:12.737355 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:12.738392 4117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:12.740056 4117 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:12.740146 4117 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:12.740265 4117 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:12.740329 4117 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:12.773759 4117 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:46143
I20240501 14:02:12.773823 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:46143 every 8 connection(s)
I20240501 14:02:12.775187 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:02:12.781690 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4117
I20240501 14:02:12.781926 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:02:12.785868 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:12.789916 4244 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:12.790310 4244 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:12.791447 4244 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:12.793954 4058 ts_manager.cc:194] Registered new tserver with Master: eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193:46143)
I20240501 14:02:12.795598 4058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:37511
W20240501 14:02:13.150126 4248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:13.150455 4248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:13.150563 4248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:13.161746 4248 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:13.161921 4248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:13.162111 4248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 14:02:13.177170 4248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:13.178540 4248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:13.180347 4248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:13.192596 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:13.193048 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:13.193507 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:13.194177 4248 server_base.cc:1034] running on GCE node
I20240501 14:02:13.194795 4248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:13.195775 4248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:13.196991 4248 hybrid_clock.cc:648] HybridClock initialized: now 1714572133196931 us; error 41 us; skew 500 ppm
I20240501 14:02:13.197345 4248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:13.199640 4248 webserver.cc:460] Webserver started at http://127.22.5.194:33159/ using document root <none> and password file <none>
I20240501 14:02:13.200379 4248 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:13.200496 4248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:13.200848 4248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:13.203569 4248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "58338e37952447739d03fd511b3c63c9"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.204349 4248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "58338e37952447739d03fd511b3c63c9"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.209832 4248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 14:02:13.213343 4262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.214730 4248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:13.214968 4248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "58338e37952447739d03fd511b3c63c9"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.215191 4248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:13.242311 4248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:13.243238 4248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:13.243554 4248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:13.244571 4248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:13.246178 4248 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:13.246309 4248 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.246433 4248 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:13.246513 4248 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.279518 4248 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:34331
I20240501 14:02:13.279577 4374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:34331 every 8 connection(s)
I20240501 14:02:13.280905 4248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:13.287137 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4248
I20240501 14:02:13.287367 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:02:13.290979 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:13.295209 4375 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:13.295552 4375 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:13.296449 4375 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:13.297920 4058 ts_manager.cc:194] Registered new tserver with Master: 58338e37952447739d03fd511b3c63c9 (127.22.5.194:34331)
I20240501 14:02:13.298682 4058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:48279
W20240501 14:02:13.653043 4378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:13.653395 4378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:13.653553 4378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:13.661279 4378 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:13.661424 4378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:13.661592 4378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:02:13.670912 4378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:13.672314 4378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:13.674124 4378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:13.686734 4384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:13.687219 4386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:13.687733 4383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:13.688438 4378 server_base.cc:1034] running on GCE node
I20240501 14:02:13.689059 4378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:13.690073 4378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:13.691273 4378 hybrid_clock.cc:648] HybridClock initialized: now 1714572133691226 us; error 39 us; skew 500 ppm
I20240501 14:02:13.691609 4378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:13.694116 4378 webserver.cc:460] Webserver started at http://127.22.5.195:46069/ using document root <none> and password file <none>
I20240501 14:02:13.694905 4378 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:13.695021 4378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:13.695343 4378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:13.698187 4378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cec76a39f06a4b518ddf8c287326bd5a"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.699003 4378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cec76a39f06a4b518ddf8c287326bd5a"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.704739 4378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:02:13.708355 4392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.709957 4378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:02:13.710112 4378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cec76a39f06a4b518ddf8c287326bd5a"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:13.710300 4378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:13.743845 4378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:13.745054 4378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:13.745465 4378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:13.746533 4378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:13.748214 4378 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:13.748309 4378 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.748402 4378 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:13.748471 4378 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:13.782780 4378 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:34671
I20240501 14:02:13.782846 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:34671 every 8 connection(s)
I20240501 14:02:13.784113 4378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:13.790292 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4378
I20240501 14:02:13.790489 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:02:13.794085 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:13.797631 4505 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:13.797986 4505 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:13.799063 4505 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:13.799338 4244 heartbeater.cc:503] Master 127.22.5.254:42339 was elected leader, sending a full tablet report...
I20240501 14:02:13.800663 4057 ts_manager.cc:194] Registered new tserver with Master: cec76a39f06a4b518ddf8c287326bd5a (127.22.5.195:34671)
I20240501 14:02:13.801630 4057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:39053
W20240501 14:02:14.148068 4508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:14.148435 4508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:14.148595 4508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:14.156124 4508 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:14.156306 4508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:14.156486 4508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 14:02:14.165645 4508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:14.166990 4508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:14.168699 4508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:14.180969 4513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:14.181460 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:14.181900 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:14.182519 4508 server_base.cc:1034] running on GCE node
I20240501 14:02:14.183097 4508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:14.184051 4508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:14.185261 4508 hybrid_clock.cc:648] HybridClock initialized: now 1714572134185179 us; error 39 us; skew 500 ppm
I20240501 14:02:14.185595 4508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:14.191041 4508 webserver.cc:460] Webserver started at http://127.22.5.196:37155/ using document root <none> and password file <none>
I20240501 14:02:14.191880 4508 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:14.192003 4508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:14.192385 4508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:14.195111 4508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "deb57b3ec55e4dd191f53edd332c5eaf"
format_stamp: "Formatted at 2024-05-01 14:02:14 on dist-test-slave-5jzj"
I20240501 14:02:14.195972 4508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "deb57b3ec55e4dd191f53edd332c5eaf"
format_stamp: "Formatted at 2024-05-01 14:02:14 on dist-test-slave-5jzj"
I20240501 14:02:14.201661 4508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 14:02:14.205169 4522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:14.206602 4508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:14.206773 4508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "deb57b3ec55e4dd191f53edd332c5eaf"
format_stamp: "Formatted at 2024-05-01 14:02:14 on dist-test-slave-5jzj"
I20240501 14:02:14.206985 4508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:14.224936 4508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:14.225862 4508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:14.226138 4508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:14.227133 4508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:14.228657 4508 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:14.228755 4508 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:14.228870 4508 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:14.228941 4508 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:14.260207 4508 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:33445
I20240501 14:02:14.260262 4634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:33445 every 8 connection(s)
I20240501 14:02:14.261647 4508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:14.263017 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4508
I20240501 14:02:14.263226 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:02:14.274536 4635 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:14.274817 4635 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:14.275632 4635 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:14.276979 4057 ts_manager.cc:194] Registered new tserver with Master: deb57b3ec55e4dd191f53edd332c5eaf (127.22.5.196:33445)
I20240501 14:02:14.277863 4057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:41885
I20240501 14:02:14.279695 22551 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:02:14.301087 4375 heartbeater.cc:503] Master 127.22.5.254:42339 was elected leader, sending a full tablet report...
I20240501 14:02:14.297578 4057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41324:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:02:14.327401 4310 tablet_service.cc:1467] Processing CreateTablet for tablet 177ac6f6737944cabefd4be6af04b218 (DEFAULT_TABLE table=TestTable [id=a21692688ae84133bda7304907f87903]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.328670 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177ac6f6737944cabefd4be6af04b218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.338603 4649 tablet_bootstrap.cc:492] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:14.341778 4649 tablet_bootstrap.cc:654] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.342783 4649 log.cc:824] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:14.345248 4649 tablet_bootstrap.cc:492] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:14.345505 4649 ts_tablet_manager.cc:1367] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 14:02:14.351500 4649 raft_consensus.cc:348] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:14.351840 4649 raft_consensus.cc:374] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.351934 4649 raft_consensus.cc:729] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.352514 4649 consensus_queue.cc:260] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:14.352722 4649 raft_consensus.cc:388] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:14.352840 4649 raft_consensus.cc:482] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:14.352985 4649 raft_consensus.cc:3037] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.354972 4649 raft_consensus.cc:504] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:14.355446 4649 leader_election.cc:302] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:14.356106 4649 leader_election.cc:288] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:14.356324 4651 raft_consensus.cc:2781] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:14.356966 4651 raft_consensus.cc:686] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:14.357586 4651 consensus_queue.cc:237] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:14.358006 4649 ts_tablet_manager.cc:1398] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20240501 14:02:14.362874 4057 catalog_manager.cc:5484] T 177ac6f6737944cabefd4be6af04b218 P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:14.372645 22551 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:02:14.374742 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver eb9930dcb1e94a7ea471bd95baa3c33d to finish bootstrapping
I20240501 14:02:14.383647 22551 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 58338e37952447739d03fd511b3c63c9 to finish bootstrapping
I20240501 14:02:14.390906 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cec76a39f06a4b518ddf8c287326bd5a to finish bootstrapping
I20240501 14:02:14.399430 22551 ts_itest-base.cc:233] Waiting for 0 tablets on tserver deb57b3ec55e4dd191f53edd332c5eaf to finish bootstrapping
I20240501 14:02:14.408668 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4378
I20240501 14:02:14.409682 4500 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:14.489727 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4378
I20240501 14:02:14.506768 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4508
I20240501 14:02:14.508117 4630 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:14.589066 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4508
I20240501 14:02:18.357792 4058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41324:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:18.377343 4177 tablet_service.cc:1467] Processing CreateTablet for tablet 04358cdaac7a4293b00e7bfb108dd57e (DEFAULT_TABLE table=rebalance_test_table_0 [id=776e895b8f8f4ba79156e392250f3b9e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.377275 4179 tablet_service.cc:1467] Processing CreateTablet for tablet 92c3d3bd6056493aad9c6aa9ffc5a70c (DEFAULT_TABLE table=rebalance_test_table_0 [id=776e895b8f8f4ba79156e392250f3b9e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.379027 4179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c3d3bd6056493aad9c6aa9ffc5a70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.379492 4177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04358cdaac7a4293b00e7bfb108dd57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.377300 4178 tablet_service.cc:1467] Processing CreateTablet for tablet 184fb827a43c4c3ebb29b194f2cfc567 (DEFAULT_TABLE table=rebalance_test_table_0 [id=776e895b8f8f4ba79156e392250f3b9e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.380931 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 184fb827a43c4c3ebb29b194f2cfc567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.391603 4668 tablet_bootstrap.cc:492] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:18.394575 4668 tablet_bootstrap.cc:654] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.395593 4668 log.cc:824] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:18.398155 4668 tablet_bootstrap.cc:492] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:18.398381 4668 ts_tablet_manager.cc:1367] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:02:18.404600 4668 raft_consensus.cc:348] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.404958 4668 raft_consensus.cc:374] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.405058 4668 raft_consensus.cc:729] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.405777 4668 consensus_queue.cc:260] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.405997 4668 raft_consensus.cc:388] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.406128 4668 raft_consensus.cc:482] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.406277 4668 raft_consensus.cc:3037] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.408275 4668 raft_consensus.cc:504] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.408800 4668 leader_election.cc:302] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:18.409498 4668 leader_election.cc:288] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.409633 4670 raft_consensus.cc:2781] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.410694 4670 raft_consensus.cc:686] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:18.411332 4670 consensus_queue.cc:237] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.411559 4668 ts_tablet_manager.cc:1398] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240501 14:02:18.411955 4668 tablet_bootstrap.cc:492] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:18.413692 4668 tablet_bootstrap.cc:654] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.415755 4668 tablet_bootstrap.cc:492] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:18.415906 4668 ts_tablet_manager.cc:1367] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:02:18.416671 4668 raft_consensus.cc:348] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.416807 4668 raft_consensus.cc:374] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.416669 4058 catalog_manager.cc:5484] T 92c3d3bd6056493aad9c6aa9ffc5a70c P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:18.416880 4668 raft_consensus.cc:729] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.417099 4668 consensus_queue.cc:260] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.417307 4668 raft_consensus.cc:388] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.417385 4668 raft_consensus.cc:482] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.417493 4668 raft_consensus.cc:3037] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.419757 4668 raft_consensus.cc:504] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.420043 4668 leader_election.cc:302] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:18.420325 4668 leader_election.cc:288] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.420408 4670 raft_consensus.cc:2781] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.420588 4670 raft_consensus.cc:686] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:18.420774 4670 consensus_queue.cc:237] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.420938 4668 ts_tablet_manager.cc:1398] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:18.421273 4668 tablet_bootstrap.cc:492] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:18.423058 4668 tablet_bootstrap.cc:654] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.423642 4058 catalog_manager.cc:5484] T 04358cdaac7a4293b00e7bfb108dd57e P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:18.424590 4668 tablet_bootstrap.cc:492] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:18.424734 4668 ts_tablet_manager.cc:1367] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:18.425273 4668 raft_consensus.cc:348] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.425431 4668 raft_consensus.cc:374] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.425519 4668 raft_consensus.cc:729] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.425737 4668 consensus_queue.cc:260] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.425921 4668 raft_consensus.cc:388] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.426023 4668 raft_consensus.cc:482] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.426137 4668 raft_consensus.cc:3037] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.428014 4668 raft_consensus.cc:504] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.428206 4668 leader_election.cc:302] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:18.428390 4668 leader_election.cc:288] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.428467 4670 raft_consensus.cc:2781] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.428653 4670 raft_consensus.cc:686] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:18.428828 4668 ts_tablet_manager.cc:1398] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:02:18.428874 4670 consensus_queue.cc:237] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:18.430797 4058 catalog_manager.cc:5484] T 184fb827a43c4c3ebb29b194f2cfc567 P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:18.953258 4058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41324:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:18.962136 4309 tablet_service.cc:1467] Processing CreateTablet for tablet e47c517250544a80a50737324407f872 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a0defa2ac796483e9febfd85afff2a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.962208 4310 tablet_service.cc:1467] Processing CreateTablet for tablet 55fd7e10770b42a2bedfd3ec7df6260b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a0defa2ac796483e9febfd85afff2a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.962235 4308 tablet_service.cc:1467] Processing CreateTablet for tablet d72dd0f37e21439ab5ceda561b6ba86a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a0defa2ac796483e9febfd85afff2a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.962646 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47c517250544a80a50737324407f872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.962852 4308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72dd0f37e21439ab5ceda561b6ba86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.963100 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fd7e10770b42a2bedfd3ec7df6260b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.968343 4702 tablet_bootstrap.cc:492] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:18.970343 4702 tablet_bootstrap.cc:654] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.972118 4702 tablet_bootstrap.cc:492] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:18.972265 4702 ts_tablet_manager.cc:1367] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:18.973057 4702 raft_consensus.cc:348] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.973249 4702 raft_consensus.cc:374] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.973322 4702 raft_consensus.cc:729] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.973532 4702 consensus_queue.cc:260] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.973671 4702 raft_consensus.cc:388] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.973727 4702 raft_consensus.cc:482] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.973801 4702 raft_consensus.cc:3037] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.975631 4702 raft_consensus.cc:504] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.975857 4702 leader_election.cc:302] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:18.976261 4702 leader_election.cc:288] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.976438 4704 raft_consensus.cc:2781] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.977007 4702 ts_tablet_manager.cc:1398] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:18.977003 4704 raft_consensus.cc:686] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:18.977316 4702 tablet_bootstrap.cc:492] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:18.977404 4704 consensus_queue.cc:237] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.979161 4702 tablet_bootstrap.cc:654] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.979976 4058 catalog_manager.cc:5484] T e47c517250544a80a50737324407f872 P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:18.982355 4702 tablet_bootstrap.cc:492] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:18.982515 4702 ts_tablet_manager.cc:1367] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:02:18.983218 4702 raft_consensus.cc:348] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.983402 4702 raft_consensus.cc:374] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.983490 4702 raft_consensus.cc:729] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.983721 4702 consensus_queue.cc:260] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.983865 4702 raft_consensus.cc:388] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.983973 4702 raft_consensus.cc:482] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.984079 4702 raft_consensus.cc:3037] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.986423 4702 raft_consensus.cc:504] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.986620 4702 leader_election.cc:302] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:18.986783 4702 leader_election.cc:288] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.986853 4704 raft_consensus.cc:2781] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.987025 4704 raft_consensus.cc:686] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:18.987250 4704 consensus_queue.cc:237] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.987414 4702 ts_tablet_manager.cc:1398] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240501 14:02:18.987728 4702 tablet_bootstrap.cc:492] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:18.989311 4058 catalog_manager.cc:5484] T d72dd0f37e21439ab5ceda561b6ba86a P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:18.989771 4702 tablet_bootstrap.cc:654] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.991530 4702 tablet_bootstrap.cc:492] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:18.991703 4702 ts_tablet_manager.cc:1367] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:18.992425 4702 raft_consensus.cc:348] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.992627 4702 raft_consensus.cc:374] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.992731 4702 raft_consensus.cc:729] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.992924 4702 consensus_queue.cc:260] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.993089 4702 raft_consensus.cc:388] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:18.993218 4702 raft_consensus.cc:482] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:18.993345 4702 raft_consensus.cc:3037] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.995275 4702 raft_consensus.cc:504] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.995581 4702 leader_election.cc:302] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:18.995777 4702 leader_election.cc:288] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:18.995851 4706 raft_consensus.cc:2781] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.996012 4706 raft_consensus.cc:686] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:18.996299 4702 ts_tablet_manager.cc:1398] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:18.996332 4706 consensus_queue.cc:237] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:18.998623 4057 catalog_manager.cc:5484] T 55fd7e10770b42a2bedfd3ec7df6260b P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:19.529405 4057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41324:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:19.538301 4178 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca7b3db740b4e08a87942926f3c7e36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e01e3c9fbb044a129fe9169ce41f545b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.538587 4179 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9e837a2b6d457492e203754e42678c (DEFAULT_TABLE table=rebalance_test_table_2 [id=e01e3c9fbb044a129fe9169ce41f545b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.538775 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca7b3db740b4e08a87942926f3c7e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.538777 4177 tablet_service.cc:1467] Processing CreateTablet for tablet 7ead651c35b94c1097100608d9488987 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e01e3c9fbb044a129fe9169ce41f545b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.539037 4179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9e837a2b6d457492e203754e42678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.539384 4177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ead651c35b94c1097100608d9488987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.544257 4737 tablet_bootstrap.cc:492] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:19.545861 4737 tablet_bootstrap.cc:654] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.547657 4737 tablet_bootstrap.cc:492] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:19.547778 4737 ts_tablet_manager.cc:1367] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:02:19.548357 4737 raft_consensus.cc:348] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.548496 4737 raft_consensus.cc:374] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.548568 4737 raft_consensus.cc:729] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.548789 4737 consensus_queue.cc:260] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.548935 4737 raft_consensus.cc:388] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:19.549015 4737 raft_consensus.cc:482] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:19.549139 4737 raft_consensus.cc:3037] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.551267 4737 raft_consensus.cc:504] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.551478 4737 leader_election.cc:302] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:19.551853 4737 leader_election.cc:288] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:19.552000 4739 raft_consensus.cc:2781] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.552549 4737 ts_tablet_manager.cc:1398] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 14:02:19.552795 4737 tablet_bootstrap.cc:492] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:19.552939 4739 raft_consensus.cc:686] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:19.553242 4739 consensus_queue.cc:237] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.554345 4737 tablet_bootstrap.cc:654] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.556492 4737 tablet_bootstrap.cc:492] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:19.556623 4737 ts_tablet_manager.cc:1367] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:19.556633 4057 catalog_manager.cc:5484] T 9ca7b3db740b4e08a87942926f3c7e36 P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:19.557274 4737 raft_consensus.cc:348] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.557449 4737 raft_consensus.cc:374] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.557526 4737 raft_consensus.cc:729] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.557747 4737 consensus_queue.cc:260] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.557883 4737 raft_consensus.cc:388] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:19.557972 4737 raft_consensus.cc:482] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:19.558060 4737 raft_consensus.cc:3037] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.560418 4737 raft_consensus.cc:504] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.560681 4737 leader_election.cc:302] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:19.560873 4737 leader_election.cc:288] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:19.560932 4739 raft_consensus.cc:2781] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.561264 4739 raft_consensus.cc:686] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:19.561337 4737 ts_tablet_manager.cc:1398] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:19.561609 4739 consensus_queue.cc:237] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.561719 4737 tablet_bootstrap.cc:492] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d: Bootstrap starting.
I20240501 14:02:19.563294 4737 tablet_bootstrap.cc:654] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.563740 4057 catalog_manager.cc:5484] T 5a9e837a2b6d457492e203754e42678c P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:19.564906 4737 tablet_bootstrap.cc:492] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d: No bootstrap required, opened a new log
I20240501 14:02:19.565032 4737 ts_tablet_manager.cc:1367] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:02:19.565685 4737 raft_consensus.cc:348] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.565814 4737 raft_consensus.cc:374] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.565898 4737 raft_consensus.cc:729] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.566112 4737 consensus_queue.cc:260] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.566329 4737 raft_consensus.cc:388] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:19.566443 4737 raft_consensus.cc:482] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:19.566565 4737 raft_consensus.cc:3037] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.568445 4737 raft_consensus.cc:504] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.568648 4737 leader_election.cc:302] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9930dcb1e94a7ea471bd95baa3c33d; no voters:
I20240501 14:02:19.568807 4737 leader_election.cc:288] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:19.568907 4739 raft_consensus.cc:2781] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.569054 4739 raft_consensus.cc:686] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [term 1 LEADER]: Becoming Leader. State: Replica: eb9930dcb1e94a7ea471bd95baa3c33d, State: Running, Role: LEADER
I20240501 14:02:19.569293 4737 ts_tablet_manager.cc:1398] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20240501 14:02:19.569314 4739 consensus_queue.cc:237] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } }
I20240501 14:02:19.571056 4057 catalog_manager.cc:5484] T 7ead651c35b94c1097100608d9488987 P eb9930dcb1e94a7ea471bd95baa3c33d reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9930dcb1e94a7ea471bd95baa3c33d (127.22.5.193). New cstate: current_term: 1 leader_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9930dcb1e94a7ea471bd95baa3c33d" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:20.080835 4058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41324:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:20.089852 4310 tablet_service.cc:1467] Processing CreateTablet for tablet 06ef5a3d4bcd42f4b8eed17a477ae30c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bbf354fa53124ffa826c04852c78ab86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.089977 4308 tablet_service.cc:1467] Processing CreateTablet for tablet 81e030a28e144de99d6fe4d66f330333 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bbf354fa53124ffa826c04852c78ab86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.090199 4309 tablet_service.cc:1467] Processing CreateTablet for tablet a120cddbc4374b209a1f57cb683e0f0e (DEFAULT_TABLE table=rebalance_test_table_3 [id=bbf354fa53124ffa826c04852c78ab86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.090353 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ef5a3d4bcd42f4b8eed17a477ae30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.090528 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a120cddbc4374b209a1f57cb683e0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.090847 4308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e030a28e144de99d6fe4d66f330333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.097920 4772 tablet_bootstrap.cc:492] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:20.099180 4772 tablet_bootstrap.cc:654] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.101053 4772 tablet_bootstrap.cc:492] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:20.101240 4772 ts_tablet_manager.cc:1367] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:02:20.101892 4772 raft_consensus.cc:348] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.102052 4772 raft_consensus.cc:374] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.102151 4772 raft_consensus.cc:729] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.102365 4772 consensus_queue.cc:260] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.102526 4772 raft_consensus.cc:388] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:20.102597 4772 raft_consensus.cc:482] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:20.102705 4772 raft_consensus.cc:3037] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.104559 4772 raft_consensus.cc:504] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.104821 4772 leader_election.cc:302] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:20.108333 4772 leader_election.cc:288] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:20.108498 4774 raft_consensus.cc:2781] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.109001 4774 raft_consensus.cc:686] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:20.109299 4772 ts_tablet_manager.cc:1398] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20240501 14:02:20.109440 4774 consensus_queue.cc:237] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.109570 4772 tablet_bootstrap.cc:492] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:20.111409 4772 tablet_bootstrap.cc:654] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.112025 4057 catalog_manager.cc:5484] T 81e030a28e144de99d6fe4d66f330333 P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:20.113433 4772 tablet_bootstrap.cc:492] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:20.113572 4772 ts_tablet_manager.cc:1367] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:02:20.114279 4772 raft_consensus.cc:348] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.114447 4772 raft_consensus.cc:374] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.114524 4772 raft_consensus.cc:729] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.114800 4772 consensus_queue.cc:260] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.114992 4772 raft_consensus.cc:388] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:20.115099 4772 raft_consensus.cc:482] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:20.115190 4772 raft_consensus.cc:3037] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.117625 4772 raft_consensus.cc:504] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.117884 4772 leader_election.cc:302] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:20.118188 4774 raft_consensus.cc:2781] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.118357 4774 raft_consensus.cc:686] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:20.118575 4774 consensus_queue.cc:237] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.119323 4772 leader_election.cc:288] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:20.119843 4772 ts_tablet_manager.cc:1398] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:02:20.120142 4772 tablet_bootstrap.cc:492] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9: Bootstrap starting.
I20240501 14:02:20.120651 4057 catalog_manager.cc:5484] T a120cddbc4374b209a1f57cb683e0f0e P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:20.121572 4772 tablet_bootstrap.cc:654] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.123268 4772 tablet_bootstrap.cc:492] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9: No bootstrap required, opened a new log
I20240501 14:02:20.123421 4772 ts_tablet_manager.cc:1367] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:20.124202 4772 raft_consensus.cc:348] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.124384 4772 raft_consensus.cc:374] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.124502 4772 raft_consensus.cc:729] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.124750 4772 consensus_queue.cc:260] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.124966 4772 raft_consensus.cc:388] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:20.125051 4772 raft_consensus.cc:482] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:20.125198 4772 raft_consensus.cc:3037] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.127288 4772 raft_consensus.cc:504] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.127542 4772 leader_election.cc:302] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58338e37952447739d03fd511b3c63c9; no voters:
I20240501 14:02:20.127797 4772 leader_election.cc:288] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:20.127954 4774 raft_consensus.cc:2781] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.128110 4774 raft_consensus.cc:686] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [term 1 LEADER]: Becoming Leader. State: Replica: 58338e37952447739d03fd511b3c63c9, State: Running, Role: LEADER
I20240501 14:02:20.128284 4772 ts_tablet_manager.cc:1398] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:20.128346 4774 consensus_queue.cc:237] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } }
I20240501 14:02:20.130482 4057 catalog_manager.cc:5484] T 06ef5a3d4bcd42f4b8eed17a477ae30c P 58338e37952447739d03fd511b3c63c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58338e37952447739d03fd511b3c63c9 (127.22.5.194). New cstate: current_term: 1 leader_uuid: "58338e37952447739d03fd511b3c63c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58338e37952447739d03fd511b3c63c9" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 34331 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:20.669301 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:34671
--local_ip_for_outbound_sockets=127.22.5.195
--tserver_master_addrs=127.22.5.254:42339
--webserver_port=46069
--webserver_interface=127.22.5.195
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:02:21.034122 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:21.034438 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:21.034533 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:21.042373 4804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:21.042507 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:21.042616 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:02:21.051805 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:34671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=46069
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:21.053145 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:21.054917 4804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:21.066995 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:21.067523 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:21.067925 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:21.068604 4804 server_base.cc:1034] running on GCE node
I20240501 14:02:21.069274 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:21.070223 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:21.071430 4804 hybrid_clock.cc:648] HybridClock initialized: now 1714572141071371 us; error 44 us; skew 500 ppm
I20240501 14:02:21.071815 4804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:21.074198 4804 webserver.cc:460] Webserver started at http://127.22.5.195:46069/ using document root <none> and password file <none>
I20240501 14:02:21.075031 4804 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:21.075165 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:21.080626 4804 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:21.083453 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.084784 4804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:21.084947 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cec76a39f06a4b518ddf8c287326bd5a"
format_stamp: "Formatted at 2024-05-01 14:02:13 on dist-test-slave-5jzj"
I20240501 14:02:21.085839 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:21.114364 4804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:21.115276 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:21.115571 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:21.116528 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:21.118178 4804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:21.118259 4804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.118402 4804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:21.118458 4804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.150442 4804 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:34671
I20240501 14:02:21.150487 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:34671 every 8 connection(s)
I20240501 14:02:21.152026 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:21.156734 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4804
I20240501 14:02:21.157754 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:33445
--local_ip_for_outbound_sockets=127.22.5.196
--tserver_master_addrs=127.22.5.254:42339
--webserver_port=37155
--webserver_interface=127.22.5.196
--builtin_ntp_servers=127.22.5.212:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:21.165831 4931 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:21.166188 4931 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:21.167050 4931 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:21.168660 4057 ts_manager.cc:194] Re-registered known tserver with Master: cec76a39f06a4b518ddf8c287326bd5a (127.22.5.195:34671)
I20240501 14:02:21.169529 4057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:51117
W20240501 14:02:21.521024 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:21.521404 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:21.521560 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:21.529148 4935 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:21.529314 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:21.529484 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 14:02:21.538920 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:33445
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=37155
--tserver_master_addrs=127.22.5.254:42339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:21.540417 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:21.542317 4935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:21.555208 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:21.555719 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:21.556176 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:21.556784 4935 server_base.cc:1034] running on GCE node
I20240501 14:02:21.557507 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:21.558516 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:21.559716 4935 hybrid_clock.cc:648] HybridClock initialized: now 1714572141559666 us; error 43 us; skew 500 ppm
I20240501 14:02:21.560057 4935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:21.562541 4935 webserver.cc:460] Webserver started at http://127.22.5.196:37155/ using document root <none> and password file <none>
I20240501 14:02:21.563331 4935 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:21.563459 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:21.569445 4935 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:02:21.573354 4949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.574851 4935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:21.575021 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "deb57b3ec55e4dd191f53edd332c5eaf"
format_stamp: "Formatted at 2024-05-01 14:02:14 on dist-test-slave-5jzj"
I20240501 14:02:21.575912 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:21.598148 4935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:21.599042 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:21.599334 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:21.600369 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:21.602128 4935 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:21.602203 4935 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.602334 4935 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:21.602384 4935 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:21.634820 4935 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:33445
I20240501 14:02:21.634862 5061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:33445 every 8 connection(s)
I20240501 14:02:21.636494 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:21.641266 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 4935
I20240501 14:02:21.651170 5062 heartbeater.cc:347] Connected to a master server at 127.22.5.254:42339
I20240501 14:02:21.651542 5062 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:21.652668 5062 heartbeater.cc:511] Master 127.22.5.254:42339 requested a full tablet report, sending...
I20240501 14:02:21.654382 4057 ts_manager.cc:194] Re-registered known tserver with Master: deb57b3ec55e4dd191f53edd332c5eaf (127.22.5.196:33445)
I20240501 14:02:21.655556 4057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:44331
I20240501 14:02:22.165968 4308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:22.170200 4177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:22.172444 4931 heartbeater.cc:503] Master 127.22.5.254:42339 was elected leader, sending a full tablet report...
I20240501 14:02:22.172606 4866 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.174393 4997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.338824 4997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.338850 4866 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.346041 4308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:22.351835 4177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:22.595757 4866 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.596488 4308 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:02:22.603024 4997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:22.613174 4177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
8881fbbc87b3422095055ffd66449d63 | 127.22.5.254:42339 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:41585 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
58338e37952447739d03fd511b3c63c9 | 127.22.5.194:34331 | HEALTHY | <none> | 7 | 0
cec76a39f06a4b518ddf8c287326bd5a | 127.22.5.195:34671 | HEALTHY | <none> | 0 | 0
deb57b3ec55e4dd191f53edd332c5eaf | 127.22.5.196:33445 | HEALTHY | <none> | 0 | 0
eb9930dcb1e94a7ea471bd95baa3c33d | 127.22.5.193:46143 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.22.5.193 | experimental | 127.22.5.193:46143
local_ip_for_outbound_sockets | 127.22.5.194 | experimental | 127.22.5.194:34331
local_ip_for_outbound_sockets | 127.22.5.195 | experimental | 127.22.5.195:34671
local_ip_for_outbound_sockets | 127.22.5.196 | experimental | 127.22.5.196:33445
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.5.193:46143
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.5.194:34331
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.5.195:34671
server_dump_info_path | /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.5.196:33445
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.5.212:41585 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:02:22.658275 5062 heartbeater.cc:503] Master 127.22.5.254:42339 was elected leader, sending a full tablet report...
I20240501 14:02:22.661482 22551 log_verifier.cc:126] Checking tablet 04358cdaac7a4293b00e7bfb108dd57e
I20240501 14:02:22.669656 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 04358cdaac7a4293b00e7bfb108dd57e
I20240501 14:02:22.669762 22551 log_verifier.cc:126] Checking tablet 06ef5a3d4bcd42f4b8eed17a477ae30c
I20240501 14:02:22.678252 22551 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 06ef5a3d4bcd42f4b8eed17a477ae30c
I20240501 14:02:22.678375 22551 log_verifier.cc:126] Checking tablet 177ac6f6737944cabefd4be6af04b218
I20240501 14:02:22.686371 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 177ac6f6737944cabefd4be6af04b218
I20240501 14:02:22.686475 22551 log_verifier.cc:126] Checking tablet 184fb827a43c4c3ebb29b194f2cfc567
I20240501 14:02:22.694430 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 184fb827a43c4c3ebb29b194f2cfc567
I20240501 14:02:22.694519 22551 log_verifier.cc:126] Checking tablet 55fd7e10770b42a2bedfd3ec7df6260b
I20240501 14:02:22.702378 22551 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 55fd7e10770b42a2bedfd3ec7df6260b
I20240501 14:02:22.702463 22551 log_verifier.cc:126] Checking tablet 5a9e837a2b6d457492e203754e42678c
I20240501 14:02:22.709995 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5a9e837a2b6d457492e203754e42678c
I20240501 14:02:22.710103 22551 log_verifier.cc:126] Checking tablet 7ead651c35b94c1097100608d9488987
I20240501 14:02:22.717509 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 7ead651c35b94c1097100608d9488987
I20240501 14:02:22.717607 22551 log_verifier.cc:126] Checking tablet 81e030a28e144de99d6fe4d66f330333
I20240501 14:02:22.725011 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 81e030a28e144de99d6fe4d66f330333
I20240501 14:02:22.725137 22551 log_verifier.cc:126] Checking tablet 92c3d3bd6056493aad9c6aa9ffc5a70c
I20240501 14:02:22.732753 22551 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 92c3d3bd6056493aad9c6aa9ffc5a70c
I20240501 14:02:22.732861 22551 log_verifier.cc:126] Checking tablet 9ca7b3db740b4e08a87942926f3c7e36
I20240501 14:02:22.740698 22551 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9ca7b3db740b4e08a87942926f3c7e36
I20240501 14:02:22.740803 22551 log_verifier.cc:126] Checking tablet a120cddbc4374b209a1f57cb683e0f0e
I20240501 14:02:22.748219 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a120cddbc4374b209a1f57cb683e0f0e
I20240501 14:02:22.748318 22551 log_verifier.cc:126] Checking tablet d72dd0f37e21439ab5ceda561b6ba86a
I20240501 14:02:22.755862 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d72dd0f37e21439ab5ceda561b6ba86a
I20240501 14:02:22.755966 22551 log_verifier.cc:126] Checking tablet e47c517250544a80a50737324407f872
I20240501 14:02:22.763407 22551 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e47c517250544a80a50737324407f872
I20240501 14:02:22.764345 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4117
I20240501 14:02:22.765688 4238 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:22.864099 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4117
I20240501 14:02:22.885377 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4248
I20240501 14:02:22.886435 4370 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:22.984975 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4248
I20240501 14:02:23.004678 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4804
I20240501 14:02:23.010722 4926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:23.097211 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4804
I20240501 14:02:23.115729 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4935
I20240501 14:02:23.121471 5056 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:23.208690 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4935
I20240501 14:02:23.226588 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 4027
I20240501 14:02:23.227828 4088 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:23.341429 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 4027
2024-05-01T14:02:23Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (11786 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (11786 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240501 14:02:23.398360 22551 test_util.cc:268] Using random seed: -276001353
I20240501 14:02:23.400264 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:02:23.400331 22551 ts_itest-base.cc:113] --------------
I20240501 14:02:23.400404 22551 ts_itest-base.cc:114] 4 tablet servers
I20240501 14:02:23.400454 22551 ts_itest-base.cc:115] 2 replicas per TS
I20240501 14:02:23.400521 22551 ts_itest-base.cc:116] --------------
2024-05-01T14:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:23Z Disabled control of system clock
I20240501 14:02:23.462455 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:39987
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:39987
--location_mapping_cmd=/tmp/dist-test-taskA1ASV2/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 14:02:23.810195 5175 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:02:23.810544 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:23.810657 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:23.810751 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:23.818341 5175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:23.818478 5175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:23.818559 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:23.818630 5175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:23.818724 5175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:23.828166 5175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskA1ASV2/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:39987
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:23.829593 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:23.831346 5175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:23.842547 5181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.843027 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.843503 5183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:23.844192 5175 server_base.cc:1034] running on GCE node
I20240501 14:02:23.845012 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:23.846539 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:23.847759 5175 hybrid_clock.cc:648] HybridClock initialized: now 1714572143847695 us; error 42 us; skew 500 ppm
I20240501 14:02:23.848079 5175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:23.850389 5175 webserver.cc:460] Webserver started at http://127.22.5.254:45585/ using document root <none> and password file <none>
I20240501 14:02:23.851112 5175 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:23.851241 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:23.851547 5175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:23.854269 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9299fb28aadd42b0ba60267d1889a63f"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-5jzj"
I20240501 14:02:23.855072 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9299fb28aadd42b0ba60267d1889a63f"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-5jzj"
I20240501 14:02:23.860386 5175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 14:02:23.863945 5189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:23.865347 5175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:23.865518 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9299fb28aadd42b0ba60267d1889a63f"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-5jzj"
I20240501 14:02:23.865694 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:23.890264 5175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:23.891151 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:23.891454 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:23.909562 5175 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:39987
I20240501 14:02:23.909580 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:39987 every 8 connection(s)
I20240501 14:02:23.910957 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:02:23.914695 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.918388 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 5175
I20240501 14:02:23.918584 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:02:23.925890 5241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f: Bootstrap starting.
I20240501 14:02:23.928934 5241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.930328 5241 log.cc:824] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:23.933324 5241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f: No bootstrap required, opened a new log
I20240501 14:02:23.939918 5241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } }
I20240501 14:02:23.940299 5241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.940403 5241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9299fb28aadd42b0ba60267d1889a63f, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.941043 5241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } }
I20240501 14:02:23.941300 5241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:23.941431 5241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:23.941581 5241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.943675 5241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } }
I20240501 14:02:23.944145 5241 leader_election.cc:302] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9299fb28aadd42b0ba60267d1889a63f; no voters:
I20240501 14:02:23.944751 5241 leader_election.cc:288] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:23.944905 5246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:23.945914 5246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [term 1 LEADER]: Becoming Leader. State: Replica: 9299fb28aadd42b0ba60267d1889a63f, State: Running, Role: LEADER
I20240501 14:02:23.946697 5246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } }
I20240501 14:02:23.946892 5241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:23.949761 5247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9299fb28aadd42b0ba60267d1889a63f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } } }
I20240501 14:02:23.949702 5248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9299fb28aadd42b0ba60267d1889a63f. Latest consensus state: current_term: 1 leader_uuid: "9299fb28aadd42b0ba60267d1889a63f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9299fb28aadd42b0ba60267d1889a63f" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 39987 } } }
I20240501 14:02:23.950393 5248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:23.950698 5247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:23.952598 5255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:23.958217 5255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:23.967597 5255 catalog_manager.cc:1347] Generated new cluster ID: d0a94e179af043948771978df6dbd561
I20240501 14:02:23.967728 5255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:23.983570 5255 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:23.984846 5255 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:23.992952 5255 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9299fb28aadd42b0ba60267d1889a63f: Generated new TSK 0
I20240501 14:02:23.993873 5255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:24.003697 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:02:24.384064 5265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:24.384361 5265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:24.384457 5265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:24.391922 5265 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:24.392060 5265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:24.392246 5265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 14:02:24.401595 5265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:24.402971 5265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:24.404731 5265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:24.416574 5270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.417028 5271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.417521 5273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:24.418207 5265 server_base.cc:1034] running on GCE node
I20240501 14:02:24.418730 5265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:24.419613 5265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:24.420810 5265 hybrid_clock.cc:648] HybridClock initialized: now 1714572144420769 us; error 29 us; skew 500 ppm
I20240501 14:02:24.421139 5265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:24.427269 5265 webserver.cc:460] Webserver started at http://127.22.5.193:37395/ using document root <none> and password file <none>
I20240501 14:02:24.427999 5265 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:24.428115 5265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:24.428424 5265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:24.430974 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.431814 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.436760 5265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:02:24.440129 5279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.441458 5265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:24.441604 5265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.441797 5265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:24.456122 5265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:24.456950 5265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:24.457295 5265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:24.458207 5265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:24.459594 5265 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:24.459692 5265 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.459805 5265 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:24.459877 5265 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.490834 5265 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:36725
I20240501 14:02:24.490896 5391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:36725 every 8 connection(s)
I20240501 14:02:24.492225 5265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:02:24.501550 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 5265
I20240501 14:02:24.502167 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:02:24.504859 5392 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:24.505174 5392 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:24.505771 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:24.506031 5392 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:24.555917 5206 ts_manager.cc:194] Registered new tserver with Master: cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:24.558413 5206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:55813
W20240501 14:02:24.856323 5397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:24.856628 5397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:24.856725 5397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:24.864255 5397 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:24.864377 5397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:24.864508 5397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 14:02:24.873694 5397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:24.875025 5397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:24.876762 5397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:24.889206 5405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.889683 5402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.890233 5403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:24.890780 5397 server_base.cc:1034] running on GCE node
I20240501 14:02:24.891369 5397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:24.892292 5397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:24.893491 5397 hybrid_clock.cc:648] HybridClock initialized: now 1714572144893443 us; error 39 us; skew 500 ppm
I20240501 14:02:24.893801 5397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:24.896098 5397 webserver.cc:460] Webserver started at http://127.22.5.194:46315/ using document root <none> and password file <none>
I20240501 14:02:24.896827 5397 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:24.896945 5397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:24.897317 5397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:24.899992 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5d61a4d965da4b4f988bd9cb4477f2fb"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.900765 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5d61a4d965da4b4f988bd9cb4477f2fb"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.905956 5397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:02:24.909477 5411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.911057 5397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:24.911237 5397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5d61a4d965da4b4f988bd9cb4477f2fb"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:24.911476 5397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:24.941874 5397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:24.942704 5397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:24.942938 5397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:24.943838 5397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:24.945427 5397 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:24.945523 5397 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.945605 5397 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:24.945658 5397 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.978066 5397 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:45723
I20240501 14:02:24.978152 5523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:45723 every 8 connection(s)
I20240501 14:02:24.979399 5397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:24.985148 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 5397
I20240501 14:02:24.985365 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:02:24.988817 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:24.992363 5524 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:24.992720 5524 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:24.993903 5524 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:25.062229 5206 ts_manager.cc:194] Registered new tserver with Master: 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723)
I20240501 14:02:25.063423 5206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:44509
W20240501 14:02:25.358709 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.359022 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.359175 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.366791 5528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:25.366922 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.367091 5528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:02:25.376276 5528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.377738 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.379479 5528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.391695 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.392272 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.392594 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.393355 5528 server_base.cc:1034] running on GCE node
I20240501 14:02:25.393982 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.394980 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.396226 5528 hybrid_clock.cc:648] HybridClock initialized: now 1714572145396165 us; error 45 us; skew 500 ppm
I20240501 14:02:25.396595 5528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.398947 5528 webserver.cc:460] Webserver started at http://127.22.5.195:38633/ using document root <none> and password file <none>
I20240501 14:02:25.399668 5528 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.399765 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.400025 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:25.402717 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "782e25d5016142c488de06332aa07926"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.403463 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "782e25d5016142c488de06332aa07926"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.408497 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240501 14:02:25.411796 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.413054 5528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:25.413233 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "782e25d5016142c488de06332aa07926"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.413574 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.459156 5528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.460011 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.460258 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.461264 5528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.462718 5528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.462812 5528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.462919 5528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.462990 5528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.494817 5528 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:44641
I20240501 14:02:25.494874 5654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:44641 every 8 connection(s)
I20240501 14:02:25.496964 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:25.504848 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 5528
I20240501 14:02:25.505079 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:02:25.508296 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:25.511988 5655 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:25.512300 5655 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.513070 5655 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:25.561892 5392 heartbeater.cc:503] Master 127.22.5.254:39987 was elected leader, sending a full tablet report...
I20240501 14:02:25.571730 5206 ts_manager.cc:194] Registered new tserver with Master: 782e25d5016142c488de06332aa07926 (127.22.5.195:44641)
I20240501 14:02:25.572940 5206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:44329
W20240501 14:02:25.869491 5660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.869786 5660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.869889 5660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.877302 5660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:25.877430 5660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.877552 5660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 14:02:25.886444 5660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.887729 5660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.889631 5660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.901891 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.902482 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.902907 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.903458 5660 server_base.cc:1034] running on GCE node
I20240501 14:02:25.904018 5660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.904943 5660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.906139 5660 hybrid_clock.cc:648] HybridClock initialized: now 1714572145906098 us; error 50 us; skew 500 ppm
I20240501 14:02:25.906455 5660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.908653 5660 webserver.cc:460] Webserver started at http://127.22.5.196:42655/ using document root <none> and password file <none>
I20240501 14:02:25.909636 5660 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.909765 5660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.910140 5660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:25.913527 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d060c465e0fe461da7d8063d9c3847a9"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.914465 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d060c465e0fe461da7d8063d9c3847a9"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.920791 5660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 14:02:25.924316 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.925768 5660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:25.925933 5660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d060c465e0fe461da7d8063d9c3847a9"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:25.926146 5660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.939697 5660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.940557 5660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.940826 5660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.941768 5660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.943269 5660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.943358 5660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.943475 5660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.943550 5660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.974797 5660 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:37007
I20240501 14:02:25.974848 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:37007 every 8 connection(s)
I20240501 14:02:25.976076 5660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:25.983441 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 5660
I20240501 14:02:25.983613 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:02:25.989954 5787 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:25.990228 5787 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.991010 5787 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:26.036331 5206 ts_manager.cc:194] Registered new tserver with Master: d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:26.037444 5206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:38005
I20240501 14:02:26.038472 22551 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:02:26.055343 22551 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:02:26.055506 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 5397
I20240501 14:02:26.063752 5519 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:26.066385 5524 heartbeater.cc:503] Master 127.22.5.254:39987 was elected leader, sending a full tablet report...
I20240501 14:02:26.145239 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 5397
I20240501 14:02:26.163182 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 5528
I20240501 14:02:26.171479 5650 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:26.251646 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 5528
I20240501 14:02:27.040835 5787 heartbeater.cc:503] Master 127.22.5.254:39987 was elected leader, sending a full tablet report...
I20240501 14:02:30.021850 5205 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36526:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:30.028928 5205 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 14:02:30.060362 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 390479ed72084c0eab8e9abc3a23d650 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.060011 5327 tablet_service.cc:1467] Processing CreateTablet for tablet 390479ed72084c0eab8e9abc3a23d650 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.059908 5721 tablet_service.cc:1467] Processing CreateTablet for tablet 489888747a2941d3bf4c6a7d67cc4a3c (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.059908 5326 tablet_service.cc:1467] Processing CreateTablet for tablet 489888747a2941d3bf4c6a7d67cc4a3c (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.061235 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 1f179b7c345f4d4eb7f304da148eeabb (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.061698 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489888747a2941d3bf4c6a7d67cc4a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.061691 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390479ed72084c0eab8e9abc3a23d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.060011 5720 tablet_service.cc:1467] Processing CreateTablet for tablet 1f179b7c345f4d4eb7f304da148eeabb (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:30.062176 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f179b7c345f4d4eb7f304da148eeabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.062208 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f179b7c345f4d4eb7f304da148eeabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.063251 5327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390479ed72084c0eab8e9abc3a23d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.064363 5721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489888747a2941d3bf4c6a7d67cc4a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:30.081974 5809 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:30.083427 5810 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:30.085409 5809 tablet_bootstrap.cc:654] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.086264 5810 tablet_bootstrap.cc:654] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.086549 5809 log.cc:824] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:30.087389 5810 log.cc:824] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:30.090071 5809 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:30.090363 5809 ts_tablet_manager.cc:1367] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 14:02:30.090369 5810 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:30.090605 5810 ts_tablet_manager.cc:1367] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:02:30.096632 5810 raft_consensus.cc:348] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.096632 5809 raft_consensus.cc:348] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.097036 5809 raft_consensus.cc:374] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.097050 5810 raft_consensus.cc:374] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.097186 5810 raft_consensus.cc:729] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.097177 5809 raft_consensus.cc:729] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.097813 5810 consensus_queue.cc:260] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.097831 5809 consensus_queue.cc:260] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.099838 5809 ts_tablet_manager.cc:1398] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 14:02:30.100322 5809 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:30.101400 5810 ts_tablet_manager.cc:1398] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20240501 14:02:30.101941 5810 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:30.102249 5809 tablet_bootstrap.cc:654] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.103711 5810 tablet_bootstrap.cc:654] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.105615 5809 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:30.105795 5810 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:30.105782 5809 ts_tablet_manager.cc:1367] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:30.106451 5810 ts_tablet_manager.cc:1367] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:30.107059 5809 raft_consensus.cc:348] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.107276 5809 raft_consensus.cc:374] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.107378 5809 raft_consensus.cc:729] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.107545 5810 raft_consensus.cc:348] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.107743 5810 raft_consensus.cc:374] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.107637 5809 consensus_queue.cc:260] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.107837 5810 raft_consensus.cc:729] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.108096 5810 consensus_queue.cc:260] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.108253 5809 ts_tablet_manager.cc:1398] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:30.108639 5810 ts_tablet_manager.cc:1398] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:30.108712 5809 tablet_bootstrap.cc:492] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:30.108922 5810 tablet_bootstrap.cc:492] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:30.110495 5810 tablet_bootstrap.cc:654] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.110502 5809 tablet_bootstrap.cc:654] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:30.112668 5809 tablet_bootstrap.cc:492] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:30.113201 5809 ts_tablet_manager.cc:1367] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:30.113804 5810 tablet_bootstrap.cc:492] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:30.113786 5809 raft_consensus.cc:348] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.113961 5810 ts_tablet_manager.cc:1367] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:30.114130 5809 raft_consensus.cc:374] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.114225 5809 raft_consensus.cc:729] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.114480 5809 consensus_queue.cc:260] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.114557 5810 raft_consensus.cc:348] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.114953 5809 ts_tablet_manager.cc:1398] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:30.115142 5810 raft_consensus.cc:374] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:30.115235 5810 raft_consensus.cc:729] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:30.115496 5810 consensus_queue.cc:260] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.115999 5810 ts_tablet_manager.cc:1398] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:30.193413 5813 raft_consensus.cc:482] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:30.193689 5813 raft_consensus.cc:504] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.195236 5813 leader_election.cc:288] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.201977 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f179b7c345f4d4eb7f304da148eeabb" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:30.202564 5347 raft_consensus.cc:2454] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:30.203260 5677 leader_election.cc:302] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.203624 5813 raft_consensus.cc:2781] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:30.203763 5813 raft_consensus.cc:482] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:30.203859 5813 raft_consensus.cc:3037] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.205922 5813 raft_consensus.cc:504] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.206413 5813 leader_election.cc:288] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.206965 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f179b7c345f4d4eb7f304da148eeabb" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:30.207165 5347 raft_consensus.cc:3037] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.209266 5347 raft_consensus.cc:2454] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:30.209668 5677 leader_election.cc:302] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.209959 5813 raft_consensus.cc:2781] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:30.210390 5813 raft_consensus.cc:686] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:30.210825 5813 consensus_queue.cc:237] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.215865 5205 catalog_manager.cc:5484] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } }
W20240501 14:02:30.231329 5788 tablet.cc:2369] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:30.248459 5393 tablet.cc:2369] T 1f179b7c345f4d4eb7f304da148eeabb P cc70ab919f1c47a7a648dfe19791e7e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:30.258423 5813 raft_consensus.cc:482] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:30.258639 5813 raft_consensus.cc:504] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.259155 5813 leader_election.cc:288] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.259785 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:30.260031 5347 raft_consensus.cc:2454] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:30.260416 5677 leader_election.cc:302] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.260722 5813 raft_consensus.cc:2781] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:30.260824 5813 raft_consensus.cc:482] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:30.260934 5813 raft_consensus.cc:3037] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.262799 5813 raft_consensus.cc:504] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.263231 5813 leader_election.cc:288] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.263708 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:30.263902 5347 raft_consensus.cc:3037] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.265707 5347 raft_consensus.cc:2454] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:30.266072 5677 leader_election.cc:302] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.266335 5813 raft_consensus.cc:2781] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:30.266494 5813 raft_consensus.cc:686] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:30.266729 5813 consensus_queue.cc:237] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.268936 5205 catalog_manager.cc:5484] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:30.356106 5813 raft_consensus.cc:482] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:30.356307 5813 raft_consensus.cc:504] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.356815 5813 leader_election.cc:288] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.357460 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489888747a2941d3bf4c6a7d67cc4a3c" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:30.357682 5347 raft_consensus.cc:2454] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:30.358114 5677 leader_election.cc:302] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.358397 5813 raft_consensus.cc:2781] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:30.358489 5813 raft_consensus.cc:482] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:30.358585 5813 raft_consensus.cc:3037] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.360431 5813 raft_consensus.cc:504] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.360877 5813 leader_election.cc:288] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:30.361467 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489888747a2941d3bf4c6a7d67cc4a3c" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:30.361660 5347 raft_consensus.cc:3037] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:30.363520 5347 raft_consensus.cc:2454] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:30.363934 5677 leader_election.cc:302] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:30.364230 5813 raft_consensus.cc:2781] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:30.364358 5813 raft_consensus.cc:686] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:30.364661 5813 consensus_queue.cc:237] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:30.367029 5205 catalog_manager.cc:5484] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:30.617281 5813 consensus_queue.cc:1035] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:30.678473 5813 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:30.783363 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:30.783752 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:30.791599 5824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:30.805915 5830 consensus_queue.cc:1035] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 218.475 ms
time per row: 0.109238 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:02:32.102816 5206 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36526:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:32.103623 5206 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 14:02:32.112229 5720 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0beea9410547eda2c483ece1d6da43 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.112568 5327 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0beea9410547eda2c483ece1d6da43 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.112704 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0beea9410547eda2c483ece1d6da43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.112993 5327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0beea9410547eda2c483ece1d6da43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.112939 5722 tablet_service.cc:1467] Processing CreateTablet for tablet f2f9b497b92144ab9831c75df9a1de8e (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.113353 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b497b92144ab9831c75df9a1de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.117787 5325 tablet_service.cc:1467] Processing CreateTablet for tablet f2f9b497b92144ab9831c75df9a1de8e (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.117852 5721 tablet_service.cc:1467] Processing CreateTablet for tablet 6b42621fbaf84e8fa42faa2fd37c4b51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.118216 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b497b92144ab9831c75df9a1de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.118382 5721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b42621fbaf84e8fa42faa2fd37c4b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.120501 5875 tablet_bootstrap.cc:492] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:32.122262 5875 tablet_bootstrap.cc:654] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.123785 5326 tablet_service.cc:1467] Processing CreateTablet for tablet 6b42621fbaf84e8fa42faa2fd37c4b51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:32.124230 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b42621fbaf84e8fa42faa2fd37c4b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:32.125775 5875 tablet_bootstrap.cc:492] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:32.125969 5875 ts_tablet_manager.cc:1367] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:32.126855 5875 raft_consensus.cc:348] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.127151 5875 raft_consensus.cc:374] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.127223 5877 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:32.127323 5875 raft_consensus.cc:729] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.127580 5875 consensus_queue.cc:260] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.128280 5875 ts_tablet_manager.cc:1398] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:32.128592 5875 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:32.129060 5877 tablet_bootstrap.cc:654] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.130313 5875 tablet_bootstrap.cc:654] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.132010 5877 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:32.132161 5877 ts_tablet_manager.cc:1367] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:32.132541 5875 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:32.132683 5875 ts_tablet_manager.cc:1367] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:32.132987 5877 raft_consensus.cc:348] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.133404 5875 raft_consensus.cc:348] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.133527 5877 raft_consensus.cc:374] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.133626 5875 raft_consensus.cc:374] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.133679 5877 raft_consensus.cc:729] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.133713 5875 raft_consensus.cc:729] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.133924 5877 consensus_queue.cc:260] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.134017 5875 consensus_queue.cc:260] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.134573 5875 ts_tablet_manager.cc:1398] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:32.134941 5875 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:32.135398 5877 ts_tablet_manager.cc:1398] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:32.135655 5877 tablet_bootstrap.cc:492] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:32.136328 5875 tablet_bootstrap.cc:654] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.137312 5877 tablet_bootstrap.cc:654] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.138451 5875 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:32.138579 5875 ts_tablet_manager.cc:1367] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:32.139176 5877 tablet_bootstrap.cc:492] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:32.139387 5877 ts_tablet_manager.cc:1367] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:02:32.139360 5875 raft_consensus.cc:348] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.139580 5875 raft_consensus.cc:374] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.139707 5875 raft_consensus.cc:729] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.139974 5875 consensus_queue.cc:260] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.140322 5877 raft_consensus.cc:348] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.140549 5877 raft_consensus.cc:374] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.140571 5875 ts_tablet_manager.cc:1398] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:32.140695 5877 raft_consensus.cc:729] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.140920 5877 consensus_queue.cc:260] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.141383 5877 ts_tablet_manager.cc:1398] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:32.141629 5877 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:32.143429 5877 tablet_bootstrap.cc:654] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:32.145627 5877 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:32.145788 5877 ts_tablet_manager.cc:1367] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:02:32.146498 5877 raft_consensus.cc:348] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.146672 5877 raft_consensus.cc:374] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:32.146773 5877 raft_consensus.cc:729] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:32.147018 5877 consensus_queue.cc:260] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.147593 5877 ts_tablet_manager.cc:1398] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:32.295559 5813 raft_consensus.cc:482] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:32.295823 5813 raft_consensus.cc:504] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.296388 5813 leader_election.cc:288] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:32.297062 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0beea9410547eda2c483ece1d6da43" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:32.297314 5346 raft_consensus.cc:2454] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:32.297767 5677 leader_election.cc:302] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.298053 5813 raft_consensus.cc:2781] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:32.298166 5813 raft_consensus.cc:482] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:32.298269 5813 raft_consensus.cc:3037] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.300217 5813 raft_consensus.cc:504] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.300707 5813 leader_election.cc:288] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:32.301294 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0beea9410547eda2c483ece1d6da43" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:32.301492 5346 raft_consensus.cc:3037] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.303444 5346 raft_consensus.cc:2454] T 1f0beea9410547eda2c483ece1d6da43 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:32.303839 5677 leader_election.cc:302] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.304136 5813 raft_consensus.cc:2781] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:32.304517 5813 raft_consensus.cc:686] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:32.304765 5813 consensus_queue.cc:237] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.307142 5203 catalog_manager.cc:5484] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:32.327646 5880 raft_consensus.cc:482] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:32.327877 5880 raft_consensus.cc:504] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.329277 5880 leader_election.cc:288] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:32.335515 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b497b92144ab9831c75df9a1de8e" candidate_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d060c465e0fe461da7d8063d9c3847a9" is_pre_election: true
I20240501 14:02:32.335839 5742 raft_consensus.cc:2454] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc70ab919f1c47a7a648dfe19791e7e7 in term 0.
I20240501 14:02:32.336359 5280 leader_election.cc:302] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.336728 5880 raft_consensus.cc:2781] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:32.336860 5880 raft_consensus.cc:482] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:32.336958 5880 raft_consensus.cc:3037] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.338969 5880 raft_consensus.cc:504] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.339471 5880 leader_election.cc:288] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [CANDIDATE]: Term 1 election: Requested vote from peers d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:32.340065 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f9b497b92144ab9831c75df9a1de8e" candidate_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
I20240501 14:02:32.340262 5742 raft_consensus.cc:3037] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.342325 5742 raft_consensus.cc:2454] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc70ab919f1c47a7a648dfe19791e7e7 in term 1.
I20240501 14:02:32.342746 5280 leader_election.cc:302] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.343062 5880 raft_consensus.cc:2781] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:32.343458 5880 raft_consensus.cc:686] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 LEADER]: Becoming Leader. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Running, Role: LEADER
I20240501 14:02:32.343958 5880 consensus_queue.cc:237] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.347404 5203 catalog_manager.cc:5484] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:32.451274 5813 raft_consensus.cc:482] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:32.451495 5813 raft_consensus.cc:504] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.452082 5813 leader_election.cc:288] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:32.452719 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b42621fbaf84e8fa42faa2fd37c4b51" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:32.452942 5346 raft_consensus.cc:2454] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:32.453390 5677 leader_election.cc:302] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.453656 5813 raft_consensus.cc:2781] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:32.453769 5813 raft_consensus.cc:482] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:32.453837 5813 raft_consensus.cc:3037] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.455786 5813 raft_consensus.cc:504] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.456243 5813 leader_election.cc:288] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:32.456853 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b42621fbaf84e8fa42faa2fd37c4b51" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:32.457082 5346 raft_consensus.cc:3037] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:32.459017 5346 raft_consensus.cc:2454] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:32.459445 5677 leader_election.cc:302] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:32.459728 5813 raft_consensus.cc:2781] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:32.459854 5813 raft_consensus.cc:686] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:32.460134 5813 consensus_queue.cc:237] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:32.462837 5203 catalog_manager.cc:5484] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } }
W20240501 14:02:32.862087 5891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:32.862450 5891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:32.870362 5891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:02:32.906807 5813 consensus_queue.cc:1035] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:32.919384 5907 consensus_queue.cc:1035] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:32.946105 5884 consensus_queue.cc:1035] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 214.522 ms
time per row: 0.107261 ms
I20240501 14:02:33.260444 5200 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36526:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:33.261320 5200 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 14:02:33.269840 5326 tablet_service.cc:1467] Processing CreateTablet for tablet 65e6adb2d3f14864b80432370d043b0f (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.269990 5721 tablet_service.cc:1467] Processing CreateTablet for tablet 65e6adb2d3f14864b80432370d043b0f (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.270395 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e6adb2d3f14864b80432370d043b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.270485 5721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e6adb2d3f14864b80432370d043b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.270917 5720 tablet_service.cc:1467] Processing CreateTablet for tablet aa72434772e2467580c9eda3fad83c5f (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.270972 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6cbe424b54bfa870fd79faa48f815 (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.271279 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa72434772e2467580c9eda3fad83c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.272084 5325 tablet_service.cc:1467] Processing CreateTablet for tablet aa72434772e2467580c9eda3fad83c5f (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.272509 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa72434772e2467580c9eda3fad83c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.277698 5327 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6cbe424b54bfa870fd79faa48f815 (DEFAULT_TABLE table=rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:33.278314 5327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6cbe424b54bfa870fd79faa48f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.278635 5945 tablet_bootstrap.cc:492] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:33.280158 5946 tablet_bootstrap.cc:492] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:33.280442 5945 tablet_bootstrap.cc:654] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.280906 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6cbe424b54bfa870fd79faa48f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:33.282714 5946 tablet_bootstrap.cc:654] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.289479 5946 tablet_bootstrap.cc:492] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:33.289517 5945 tablet_bootstrap.cc:492] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:33.289656 5945 ts_tablet_manager.cc:1367] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 14:02:33.289722 5946 ts_tablet_manager.cc:1367] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 14:02:33.290516 5946 raft_consensus.cc:348] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.290521 5945 raft_consensus.cc:348] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.290735 5946 raft_consensus.cc:374] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.290742 5945 raft_consensus.cc:374] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.290841 5945 raft_consensus.cc:729] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.290840 5946 raft_consensus.cc:729] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.291069 5945 consensus_queue.cc:260] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.291069 5946 consensus_queue.cc:260] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.291648 5946 ts_tablet_manager.cc:1398] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:33.291682 5945 ts_tablet_manager.cc:1398] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:33.291910 5946 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:33.291921 5945 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:33.293591 5945 tablet_bootstrap.cc:654] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.293591 5946 tablet_bootstrap.cc:654] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.298324 5946 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:33.298499 5946 ts_tablet_manager.cc:1367] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:02:33.298646 5945 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:33.298804 5945 ts_tablet_manager.cc:1367] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 14:02:33.299386 5946 raft_consensus.cc:348] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.299597 5946 raft_consensus.cc:374] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.299674 5946 raft_consensus.cc:729] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.299582 5945 raft_consensus.cc:348] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.299772 5945 raft_consensus.cc:374] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.299885 5945 raft_consensus.cc:729] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.299923 5946 consensus_queue.cc:260] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.300179 5945 consensus_queue.cc:260] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.300575 5946 ts_tablet_manager.cc:1398] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:33.300791 5945 ts_tablet_manager.cc:1398] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:33.300921 5946 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: Bootstrap starting.
I20240501 14:02:33.301102 5945 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7: Bootstrap starting.
I20240501 14:02:33.302605 5945 tablet_bootstrap.cc:654] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.302605 5946 tablet_bootstrap.cc:654] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:33.304585 5946 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: No bootstrap required, opened a new log
I20240501 14:02:33.304716 5946 ts_tablet_manager.cc:1367] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:33.304852 5945 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7: No bootstrap required, opened a new log
I20240501 14:02:33.305037 5945 ts_tablet_manager.cc:1367] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 14:02:33.305464 5946 raft_consensus.cc:348] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.305620 5946 raft_consensus.cc:374] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.305711 5946 raft_consensus.cc:729] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.305835 5945 raft_consensus.cc:348] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.305912 5946 consensus_queue.cc:260] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.306036 5945 raft_consensus.cc:374] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:33.306277 5945 raft_consensus.cc:729] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc70ab919f1c47a7a648dfe19791e7e7, State: Initialized, Role: FOLLOWER
I20240501 14:02:33.306449 5946 ts_tablet_manager.cc:1398] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:33.306507 5945 consensus_queue.cc:260] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.307134 5945 ts_tablet_manager.cc:1398] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:33.496366 5884 raft_consensus.cc:482] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:33.496628 5884 raft_consensus.cc:504] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.497398 5884 leader_election.cc:288] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.498080 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6cbe424b54bfa870fd79faa48f815" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:33.498315 5345 raft_consensus.cc:2454] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:33.498800 5677 leader_election.cc:302] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.499080 5884 raft_consensus.cc:2781] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:33.499182 5884 raft_consensus.cc:482] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:33.499284 5884 raft_consensus.cc:3037] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.502189 5884 raft_consensus.cc:504] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.502753 5884 leader_election.cc:288] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.503419 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6cbe424b54bfa870fd79faa48f815" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:33.503623 5345 raft_consensus.cc:3037] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.507541 5345 raft_consensus.cc:2454] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:33.507923 5677 leader_election.cc:302] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.508240 5884 raft_consensus.cc:2781] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:33.508424 5884 raft_consensus.cc:686] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:33.508678 5884 consensus_queue.cc:237] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.511423 5199 catalog_manager.cc:5484] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:33.522143 5884 raft_consensus.cc:482] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:33.522306 5884 raft_consensus.cc:504] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.522804 5884 leader_election.cc:288] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.523353 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e6adb2d3f14864b80432370d043b0f" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:33.523591 5345 raft_consensus.cc:2454] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:33.524016 5677 leader_election.cc:302] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.524276 5884 raft_consensus.cc:2781] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:33.524379 5884 raft_consensus.cc:482] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:33.524469 5884 raft_consensus.cc:3037] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.526427 5884 raft_consensus.cc:504] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.526871 5884 leader_election.cc:288] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.527431 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e6adb2d3f14864b80432370d043b0f" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:33.527630 5345 raft_consensus.cc:3037] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.529522 5345 raft_consensus.cc:2454] T 65e6adb2d3f14864b80432370d043b0f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:33.529917 5677 leader_election.cc:302] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.530208 5884 raft_consensus.cc:2781] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:33.530380 5884 raft_consensus.cc:686] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:33.530622 5884 consensus_queue.cc:237] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } }
I20240501 14:02:33.533309 5199 catalog_manager.cc:5484] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:33.668629 5884 raft_consensus.cc:482] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:33.668820 5884 raft_consensus.cc:504] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.669373 5884 leader_election.cc:288] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.670006 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" is_pre_election: true
I20240501 14:02:33.670217 5345 raft_consensus.cc:2454] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 0.
I20240501 14:02:33.670600 5677 leader_election.cc:302] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.670871 5884 raft_consensus.cc:2781] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:33.670954 5884 raft_consensus.cc:482] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:33.671072 5884 raft_consensus.cc:3037] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.672824 5884 raft_consensus.cc:504] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.673295 5884 leader_election.cc:288] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:33.673766 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f" candidate_uuid: "d060c465e0fe461da7d8063d9c3847a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:33.673944 5345 raft_consensus.cc:3037] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:33.675715 5345 raft_consensus.cc:2454] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d060c465e0fe461da7d8063d9c3847a9 in term 1.
I20240501 14:02:33.676133 5677 leader_election.cc:302] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cc70ab919f1c47a7a648dfe19791e7e7, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:33.676425 5884 raft_consensus.cc:2781] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:33.676575 5884 raft_consensus.cc:686] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Leader. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:33.676844 5884 consensus_queue.cc:237] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } }
I20240501 14:02:33.679206 5199 catalog_manager.cc:5484] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:33.999531 5884 consensus_queue.cc:1035] T 65e6adb2d3f14864b80432370d043b0f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:34.095791 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:34.096159 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:34.103947 5956 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:34.111615 5813 consensus_queue.cc:1035] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 14:02:34.239691 5884 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 171.62 ms
time per row: 0.0858098 ms
I20240501 14:02:34.424710 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:45723
--local_ip_for_outbound_sockets=127.22.5.194
--tserver_master_addrs=127.22.5.254:39987
--webserver_port=46315
--webserver_interface=127.22.5.194
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:02:34.778990 6002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:34.779301 6002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:34.779402 6002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:34.787082 6002 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:34.787210 6002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:34.787334 6002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 14:02:34.796814 6002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:45723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=46315
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:34.798483 6002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:34.800724 6002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:34.813570 6007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:34.814060 6008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:34.814539 6010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:34.815282 6002 server_base.cc:1034] running on GCE node
I20240501 14:02:34.815925 6002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:34.816941 6002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:34.818140 6002 hybrid_clock.cc:648] HybridClock initialized: now 1714572154818098 us; error 70 us; skew 500 ppm
I20240501 14:02:34.818466 6002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:34.820825 6002 webserver.cc:460] Webserver started at http://127.22.5.194:46315/ using document root <none> and password file <none>
I20240501 14:02:34.821612 6002 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:34.821741 6002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:34.827003 6002 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:02:34.829877 6016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:34.831384 6002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:34.831544 6002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5d61a4d965da4b4f988bd9cb4477f2fb"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-5jzj"
I20240501 14:02:34.832391 6002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:34.857693 6002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:34.858584 6002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:34.858880 6002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:34.859858 6002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:34.861445 6002 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:34.861523 6002 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:34.861651 6002 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:34.861706 6002 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:34.894320 6002 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:45723
I20240501 14:02:34.894363 6128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:45723 every 8 connection(s)
I20240501 14:02:34.895797 6002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:34.903107 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 6002
I20240501 14:02:34.903883 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:44641
--local_ip_for_outbound_sockets=127.22.5.195
--tserver_master_addrs=127.22.5.254:39987
--webserver_port=38633
--webserver_interface=127.22.5.195
--builtin_ntp_servers=127.22.5.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:34.909976 6129 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:34.910353 6129 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:34.911448 6129 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:34.913519 5197 ts_manager.cc:194] Re-registered known tserver with Master: 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723)
I20240501 14:02:34.914786 5197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:40589
W20240501 14:02:35.265076 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:35.265431 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:35.265579 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:35.273386 6133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:35.273530 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:35.273689 6133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:02:35.282963 6133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:44641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=38633
--tserver_master_addrs=127.22.5.254:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:35.284394 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:35.286201 6133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:35.298992 6139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:35.299583 6140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:35.300024 6142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:35.300678 6133 server_base.cc:1034] running on GCE node
I20240501 14:02:35.301425 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:35.302438 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:35.303643 6133 hybrid_clock.cc:648] HybridClock initialized: now 1714572155303576 us; error 54 us; skew 500 ppm
I20240501 14:02:35.303984 6133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:35.306334 6133 webserver.cc:460] Webserver started at http://127.22.5.195:38633/ using document root <none> and password file <none>
I20240501 14:02:35.307183 6133 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:35.307301 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:35.312678 6133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:35.315490 6148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:35.317023 6133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:35.317212 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "782e25d5016142c488de06332aa07926"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-5jzj"
I20240501 14:02:35.318038 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:35.334553 6133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:35.335390 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:35.335680 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:35.336633 6133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:35.338207 6133 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:35.338285 6133 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:35.338410 6133 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:35.338464 6133 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:35.369777 6133 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:44641
I20240501 14:02:35.369812 6260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:44641 every 8 connection(s)
I20240501 14:02:35.371322 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:35.378054 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 6133
I20240501 14:02:35.384294 6261 heartbeater.cc:347] Connected to a master server at 127.22.5.254:39987
I20240501 14:02:35.384624 6261 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:35.385511 6261 heartbeater.cc:511] Master 127.22.5.254:39987 requested a full tablet report, sending...
I20240501 14:02:35.387115 5197 ts_manager.cc:194] Re-registered known tserver with Master: 782e25d5016142c488de06332aa07926 (127.22.5.195:44641)
I20240501 14:02:35.388291 5197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:42157
I20240501 14:02:35.882638 6196 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.884934 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:35.890487 6064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.900038 5722 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:35.918166 6129 heartbeater.cc:503] Master 127.22.5.254:39987 was elected leader, sending a full tablet report...
I20240501 14:02:36.068928 6064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.070869 6196 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.078460 5722 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:36.087239 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:36.235162 6196 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.235785 6064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.242542 5722 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:36.260295 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:36.391247 6261 heartbeater.cc:503] Master 127.22.5.254:39987 was elected leader, sending a full tablet report...
I20240501 14:02:36.409446 6064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.416788 6196 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.418645 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:36.427888 5722 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:36.493211 5742 consensus_queue.cc:237] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:36.496315 5347 raft_consensus.cc:1261] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.496943 6403 consensus_queue.cc:1035] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.500365 5990 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
W20240501 14:02:36.505331 5675 consensus_peers.cc:485] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 -> Peer 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723): Couldn't send request to peer 5d61a4d965da4b4f988bd9cb4477f2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b42621fbaf84e8fa42faa2fd37c4b51. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.504863 5347 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
I20240501 14:02:36.506837 5197 catalog_manager.cc:5484] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:36.523857 5742 consensus_queue.cc:237] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:36.526353 5347 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.526922 6403 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.529531 5990 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.530184 5347 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.535158 5201 catalog_manager.cc:5484] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:02:36.536077 5677 consensus_peers.cc:485] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 -> Peer 782e25d5016142c488de06332aa07926 (127.22.5.195:44641): Couldn't send request to peer 782e25d5016142c488de06332aa07926. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa72434772e2467580c9eda3fad83c5f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.546976 5742 consensus_queue.cc:237] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:36.549244 5347 raft_consensus.cc:1261] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:36.549360 5675 consensus_peers.cc:485] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 -> Peer 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723): Couldn't send request to peer 5d61a4d965da4b4f988bd9cb4477f2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fb6cbe424b54bfa870fd79faa48f815. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.549791 5990 consensus_queue.cc:1035] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.552366 6403 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
I20240501 14:02:36.553015 5347 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
I20240501 14:02:36.558260 5204 catalog_manager.cc:5484] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:36.569936 5347 consensus_queue.cc:237] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:36.572538 5742 raft_consensus.cc:1261] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Refusing update from remote peer cc70ab919f1c47a7a648dfe19791e7e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.573256 6347 consensus_queue.cc:1035] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.576262 6430 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.576956 5742 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
W20240501 14:02:36.578938 5282 consensus_peers.cc:485] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 -> Peer 782e25d5016142c488de06332aa07926 (127.22.5.195:44641): Couldn't send request to peer 782e25d5016142c488de06332aa07926. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2f9b497b92144ab9831c75df9a1de8e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.582686 5204 catalog_manager.cc:5484] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New cstate: current_term: 1 leader_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.594681 5742 consensus_queue.cc:237] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:36.596798 5347 raft_consensus.cc:1261] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:36.596846 5677 consensus_peers.cc:485] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 -> Peer 782e25d5016142c488de06332aa07926 (127.22.5.195:44641): Couldn't send request to peer 782e25d5016142c488de06332aa07926. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 489888747a2941d3bf4c6a7d67cc4a3c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.597342 5990 consensus_queue.cc:1035] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.599870 6403 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.600447 5347 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.604729 5197 catalog_manager.cc:5484] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 reported cstate change: config changed from index -1 to 4, NON_VOTER 782e25d5016142c488de06332aa07926 (127.22.5.195) added. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } } }
I20240501 14:02:36.615916 5742 consensus_queue.cc:237] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
W20240501 14:02:36.618211 5675 consensus_peers.cc:485] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 -> Peer 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723): Couldn't send request to peer 5d61a4d965da4b4f988bd9cb4477f2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 390479ed72084c0eab8e9abc3a23d650. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.618353 5347 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.618844 6403 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.621570 5990 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
I20240501 14:02:36.622270 5347 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } } }
I20240501 14:02:36.627255 5201 catalog_manager.cc:5484] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index -1 to 4, NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) added. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:36.872506 6477 ts_tablet_manager.cc:902] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Initiating tablet copy from peer d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:36.874394 6477 tablet_copy_client.cc:308] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37007
I20240501 14:02:36.881507 5762 tablet_copy_service.cc:140] P d060c465e0fe461da7d8063d9c3847a9: Received BeginTabletCopySession request for tablet 6b42621fbaf84e8fa42faa2fd37c4b51 from peer 5d61a4d965da4b4f988bd9cb4477f2fb ({username='slave'} at 127.22.5.194:48071)
I20240501 14:02:36.881753 5762 tablet_copy_service.cc:161] P d060c465e0fe461da7d8063d9c3847a9: Beginning new tablet copy session on tablet 6b42621fbaf84e8fa42faa2fd37c4b51 from peer 5d61a4d965da4b4f988bd9cb4477f2fb at {username='slave'} at 127.22.5.194:48071: session id = 5d61a4d965da4b4f988bd9cb4477f2fb-6b42621fbaf84e8fa42faa2fd37c4b51
I20240501 14:02:36.884692 5762 tablet_copy_source_session.cc:205] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:36.890703 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b42621fbaf84e8fa42faa2fd37c4b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:36.905195 6477 tablet_copy_client.cc:783] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:36.905596 6477 tablet_copy_client.cc:647] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:36.908577 6477 tablet_copy_client.cc:523] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:36.913970 6477 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap starting.
I20240501 14:02:36.927706 6477 log.cc:824] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:36.968540 6510 ts_tablet_manager.cc:902] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Initiating tablet copy from peer d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:36.977824 6510 tablet_copy_client.cc:308] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37007
I20240501 14:02:36.992290 6477 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:36.992843 6477 tablet_bootstrap.cc:492] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap complete.
I20240501 14:02:36.993340 6477 ts_tablet_manager.cc:1367] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent bootstrapping tablet: real 0.080s user 0.067s sys 0.008s
I20240501 14:02:36.995658 6521 ts_tablet_manager.cc:902] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Initiating tablet copy from peer d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:36.996512 5762 tablet_copy_service.cc:140] P d060c465e0fe461da7d8063d9c3847a9: Received BeginTabletCopySession request for tablet aa72434772e2467580c9eda3fad83c5f from peer 782e25d5016142c488de06332aa07926 ({username='slave'} at 127.22.5.195:38757)
I20240501 14:02:36.996743 5762 tablet_copy_service.cc:161] P d060c465e0fe461da7d8063d9c3847a9: Beginning new tablet copy session on tablet aa72434772e2467580c9eda3fad83c5f from peer 782e25d5016142c488de06332aa07926 at {username='slave'} at 127.22.5.195:38757: session id = 782e25d5016142c488de06332aa07926-aa72434772e2467580c9eda3fad83c5f
I20240501 14:02:36.999337 6477 raft_consensus.cc:348] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:36.999840 6477 raft_consensus.cc:729] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d61a4d965da4b4f988bd9cb4477f2fb, State: Initialized, Role: LEARNER
I20240501 14:02:37.000362 6477 consensus_queue.cc:260] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:37.000988 5762 tablet_copy_source_session.cc:205] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.002249 6477 ts_tablet_manager.cc:1398] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240501 14:02:37.003300 5761 tablet_copy_service.cc:342] P d060c465e0fe461da7d8063d9c3847a9: Request end of tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-6b42621fbaf84e8fa42faa2fd37c4b51 received from {username='slave'} at 127.22.5.194:48071
I20240501 14:02:37.003466 5761 tablet_copy_service.cc:433] P d060c465e0fe461da7d8063d9c3847a9: ending tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-6b42621fbaf84e8fa42faa2fd37c4b51 on tablet 6b42621fbaf84e8fa42faa2fd37c4b51 with peer 5d61a4d965da4b4f988bd9cb4477f2fb
I20240501 14:02:37.007704 6510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa72434772e2467580c9eda3fad83c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.016252 6521 tablet_copy_client.cc:308] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37007
I20240501 14:02:37.016968 5762 tablet_copy_service.cc:140] P d060c465e0fe461da7d8063d9c3847a9: Received BeginTabletCopySession request for tablet 390479ed72084c0eab8e9abc3a23d650 from peer 5d61a4d965da4b4f988bd9cb4477f2fb ({username='slave'} at 127.22.5.194:48071)
I20240501 14:02:37.017156 5762 tablet_copy_service.cc:161] P d060c465e0fe461da7d8063d9c3847a9: Beginning new tablet copy session on tablet 390479ed72084c0eab8e9abc3a23d650 from peer 5d61a4d965da4b4f988bd9cb4477f2fb at {username='slave'} at 127.22.5.194:48071: session id = 5d61a4d965da4b4f988bd9cb4477f2fb-390479ed72084c0eab8e9abc3a23d650
I20240501 14:02:37.019888 5762 tablet_copy_source_session.cc:205] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.021061 6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390479ed72084c0eab8e9abc3a23d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.023061 6510 tablet_copy_client.cc:783] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.023371 6510 tablet_copy_client.cc:647] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.025359 6510 tablet_copy_client.cc:523] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.025727 6521 tablet_copy_client.cc:783] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.025950 6521 tablet_copy_client.cc:647] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.027472 6521 tablet_copy_client.cc:523] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.028887 6510 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Bootstrap starting.
I20240501 14:02:37.029814 6521 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap starting.
I20240501 14:02:37.038697 6510 log.cc:824] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:37.060833 6477 ts_tablet_manager.cc:902] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Initiating tablet copy from peer d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:37.061357 6521 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.061939 6521 tablet_bootstrap.cc:492] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap complete.
I20240501 14:02:37.062316 6521 ts_tablet_manager.cc:1367] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent bootstrapping tablet: real 0.033s user 0.032s sys 0.000s
I20240501 14:02:37.063027 6477 tablet_copy_client.cc:308] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37007
I20240501 14:02:37.063619 6521 raft_consensus.cc:348] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:37.063858 6521 raft_consensus.cc:729] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d61a4d965da4b4f988bd9cb4477f2fb, State: Initialized, Role: LEARNER
I20240501 14:02:37.063863 5762 tablet_copy_service.cc:140] P d060c465e0fe461da7d8063d9c3847a9: Received BeginTabletCopySession request for tablet 4fb6cbe424b54bfa870fd79faa48f815 from peer 5d61a4d965da4b4f988bd9cb4477f2fb ({username='slave'} at 127.22.5.194:48071)
I20240501 14:02:37.064087 5762 tablet_copy_service.cc:161] P d060c465e0fe461da7d8063d9c3847a9: Beginning new tablet copy session on tablet 4fb6cbe424b54bfa870fd79faa48f815 from peer 5d61a4d965da4b4f988bd9cb4477f2fb at {username='slave'} at 127.22.5.194:48071: session id = 5d61a4d965da4b4f988bd9cb4477f2fb-4fb6cbe424b54bfa870fd79faa48f815
I20240501 14:02:37.064105 6521 consensus_queue.cc:260] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:37.064661 6521 ts_tablet_manager.cc:1398] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:37.066766 5762 tablet_copy_source_session.cc:205] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.067742 5762 tablet_copy_service.cc:342] P d060c465e0fe461da7d8063d9c3847a9: Request end of tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-390479ed72084c0eab8e9abc3a23d650 received from {username='slave'} at 127.22.5.194:48071
I20240501 14:02:37.067852 5762 tablet_copy_service.cc:433] P d060c465e0fe461da7d8063d9c3847a9: ending tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-390479ed72084c0eab8e9abc3a23d650 on tablet 390479ed72084c0eab8e9abc3a23d650 with peer 5d61a4d965da4b4f988bd9cb4477f2fb
I20240501 14:02:37.068149 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6cbe424b54bfa870fd79faa48f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.074540 6510 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.075170 6510 tablet_bootstrap.cc:492] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Bootstrap complete.
I20240501 14:02:37.075263 6477 tablet_copy_client.cc:783] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.075497 6477 tablet_copy_client.cc:647] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.075687 6510 ts_tablet_manager.cc:1367] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Time spent bootstrapping tablet: real 0.047s user 0.043s sys 0.005s
I20240501 14:02:37.077314 6477 tablet_copy_client.cc:523] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.080053 6477 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap starting.
I20240501 14:02:37.080483 6510 raft_consensus.cc:348] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.080878 6510 raft_consensus.cc:729] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 782e25d5016142c488de06332aa07926, State: Initialized, Role: LEARNER
I20240501 14:02:37.081470 6510 consensus_queue.cc:260] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.083030 6510 ts_tablet_manager.cc:1398] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240501 14:02:37.084004 5762 tablet_copy_service.cc:342] P d060c465e0fe461da7d8063d9c3847a9: Request end of tablet copy session 782e25d5016142c488de06332aa07926-aa72434772e2467580c9eda3fad83c5f received from {username='slave'} at 127.22.5.195:38757
I20240501 14:02:37.084144 5762 tablet_copy_service.cc:433] P d060c465e0fe461da7d8063d9c3847a9: ending tablet copy session 782e25d5016142c488de06332aa07926-aa72434772e2467580c9eda3fad83c5f on tablet aa72434772e2467580c9eda3fad83c5f with peer 782e25d5016142c488de06332aa07926
I20240501 14:02:37.094458 6510 ts_tablet_manager.cc:902] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Initiating tablet copy from peer cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:37.095614 6510 tablet_copy_client.cc:308] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:36725
I20240501 14:02:37.102180 5367 tablet_copy_service.cc:140] P cc70ab919f1c47a7a648dfe19791e7e7: Received BeginTabletCopySession request for tablet f2f9b497b92144ab9831c75df9a1de8e from peer 782e25d5016142c488de06332aa07926 ({username='slave'} at 127.22.5.195:34161)
I20240501 14:02:37.102406 5367 tablet_copy_service.cc:161] P cc70ab919f1c47a7a648dfe19791e7e7: Beginning new tablet copy session on tablet f2f9b497b92144ab9831c75df9a1de8e from peer 782e25d5016142c488de06332aa07926 at {username='slave'} at 127.22.5.195:34161: session id = 782e25d5016142c488de06332aa07926-f2f9b497b92144ab9831c75df9a1de8e
I20240501 14:02:37.104954 5367 tablet_copy_source_session.cc:205] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.106689 6510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f9b497b92144ab9831c75df9a1de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.111153 6510 tablet_copy_client.cc:783] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.111418 6510 tablet_copy_client.cc:647] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.113574 6510 tablet_copy_client.cc:523] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.114553 6477 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.114921 6477 tablet_bootstrap.cc:492] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Bootstrap complete.
I20240501 14:02:37.115172 6477 ts_tablet_manager.cc:1367] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent bootstrapping tablet: real 0.035s user 0.035s sys 0.000s
I20240501 14:02:37.115792 6477 raft_consensus.cc:348] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:37.115985 6477 raft_consensus.cc:729] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d61a4d965da4b4f988bd9cb4477f2fb, State: Initialized, Role: LEARNER
I20240501 14:02:37.116024 6510 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Bootstrap starting.
I20240501 14:02:37.116175 6477 consensus_queue.cc:260] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: NON_VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: true } }
I20240501 14:02:37.116668 6477 ts_tablet_manager.cc:1398] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:02:37.117504 5762 tablet_copy_service.cc:342] P d060c465e0fe461da7d8063d9c3847a9: Request end of tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-4fb6cbe424b54bfa870fd79faa48f815 received from {username='slave'} at 127.22.5.194:48071
I20240501 14:02:37.117650 5762 tablet_copy_service.cc:433] P d060c465e0fe461da7d8063d9c3847a9: ending tablet copy session 5d61a4d965da4b4f988bd9cb4477f2fb-4fb6cbe424b54bfa870fd79faa48f815 on tablet 4fb6cbe424b54bfa870fd79faa48f815 with peer 5d61a4d965da4b4f988bd9cb4477f2fb
I20240501 14:02:37.131831 6540 ts_tablet_manager.cc:902] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Initiating tablet copy from peer d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:37.133824 6540 tablet_copy_client.cc:308] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37007
I20240501 14:02:37.134553 5762 tablet_copy_service.cc:140] P d060c465e0fe461da7d8063d9c3847a9: Received BeginTabletCopySession request for tablet 489888747a2941d3bf4c6a7d67cc4a3c from peer 782e25d5016142c488de06332aa07926 ({username='slave'} at 127.22.5.195:38757)
I20240501 14:02:37.134702 5762 tablet_copy_service.cc:161] P d060c465e0fe461da7d8063d9c3847a9: Beginning new tablet copy session on tablet 489888747a2941d3bf4c6a7d67cc4a3c from peer 782e25d5016142c488de06332aa07926 at {username='slave'} at 127.22.5.195:38757: session id = 782e25d5016142c488de06332aa07926-489888747a2941d3bf4c6a7d67cc4a3c
I20240501 14:02:37.136674 5762 tablet_copy_source_session.cc:205] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.137935 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489888747a2941d3bf4c6a7d67cc4a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.142602 6540 tablet_copy_client.cc:783] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.142845 6540 tablet_copy_client.cc:647] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.144580 6540 tablet_copy_client.cc:523] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.147272 6540 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Bootstrap starting.
I20240501 14:02:37.149633 6510 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.150067 6510 tablet_bootstrap.cc:492] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Bootstrap complete.
I20240501 14:02:37.150307 6510 ts_tablet_manager.cc:1367] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.008s
I20240501 14:02:37.151206 6510 raft_consensus.cc:348] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.151422 6510 raft_consensus.cc:729] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 782e25d5016142c488de06332aa07926, State: Initialized, Role: LEARNER
I20240501 14:02:37.151669 6510 consensus_queue.cc:260] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.152226 6510 ts_tablet_manager.cc:1398] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:37.153184 5367 tablet_copy_service.cc:342] P cc70ab919f1c47a7a648dfe19791e7e7: Request end of tablet copy session 782e25d5016142c488de06332aa07926-f2f9b497b92144ab9831c75df9a1de8e received from {username='slave'} at 127.22.5.195:34161
I20240501 14:02:37.153391 5367 tablet_copy_service.cc:433] P cc70ab919f1c47a7a648dfe19791e7e7: ending tablet copy session 782e25d5016142c488de06332aa07926-f2f9b497b92144ab9831c75df9a1de8e on tablet f2f9b497b92144ab9831c75df9a1de8e with peer 782e25d5016142c488de06332aa07926
I20240501 14:02:37.179086 6540 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.179548 6540 tablet_bootstrap.cc:492] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Bootstrap complete.
I20240501 14:02:37.179813 6540 ts_tablet_manager.cc:1367] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20240501 14:02:37.180523 6540 raft_consensus.cc:348] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.180733 6540 raft_consensus.cc:729] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 782e25d5016142c488de06332aa07926, State: Initialized, Role: LEARNER
I20240501 14:02:37.180948 6540 consensus_queue.cc:260] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: true } }
I20240501 14:02:37.181445 6540 ts_tablet_manager.cc:1398] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:02:37.182401 5762 tablet_copy_service.cc:342] P d060c465e0fe461da7d8063d9c3847a9: Request end of tablet copy session 782e25d5016142c488de06332aa07926-489888747a2941d3bf4c6a7d67cc4a3c received from {username='slave'} at 127.22.5.195:38757
I20240501 14:02:37.182540 5762 tablet_copy_service.cc:433] P d060c465e0fe461da7d8063d9c3847a9: ending tablet copy session 782e25d5016142c488de06332aa07926-489888747a2941d3bf4c6a7d67cc4a3c on tablet 489888747a2941d3bf4c6a7d67cc4a3c with peer 782e25d5016142c488de06332aa07926
I20240501 14:02:37.291944 6084 raft_consensus.cc:1203] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.423987 6216 raft_consensus.cc:1203] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.520210 6216 raft_consensus.cc:1203] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.579684 6084 raft_consensus.cc:1203] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.611161 6216 raft_consensus.cc:1203] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.654142 6084 raft_consensus.cc:1203] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.675451 6577 raft_consensus.cc:1050] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9: attempting to promote NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb to VOTER
I20240501 14:02:37.676240 6577 consensus_queue.cc:237] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:37.678572 5347 raft_consensus.cc:1261] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.678696 6084 raft_consensus.cc:1261] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.679193 6576 consensus_queue.cc:1035] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.679504 5990 consensus_queue.cc:1035] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.682327 6576 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.683135 5347 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.684183 6084 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.687155 5201 catalog_manager.cc:5484] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:37.694284 5742 consensus_queue.cc:237] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:37.695856 6084 raft_consensus.cc:1261] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:37.696588 6576 consensus_queue.cc:1035] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:37.699288 6577 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.701572 6084 raft_consensus.cc:2932] T 6b42621fbaf84e8fa42faa2fd37c4b51 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.703708 5190 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b42621fbaf84e8fa42faa2fd37c4b51 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:37.704406 5201 catalog_manager.cc:5484] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:37.710597 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b42621fbaf84e8fa42faa2fd37c4b51 with delete_type TABLET_DATA_TOMBSTONED (TS cc70ab919f1c47a7a648dfe19791e7e7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55334
I20240501 14:02:37.711877 6622 tablet_replica.cc:330] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: stopping tablet replica
I20240501 14:02:37.712527 6622 raft_consensus.cc:2229] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:37.712953 6622 raft_consensus.cc:2258] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:37.714900 6622 ts_tablet_manager.cc:1875] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:37.721269 6622 ts_tablet_manager.cc:1888] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:37.721472 6622 log.cc:1196] T 6b42621fbaf84e8fa42faa2fd37c4b51 P cc70ab919f1c47a7a648dfe19791e7e7: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b42621fbaf84e8fa42faa2fd37c4b51
I20240501 14:02:37.722517 5192 catalog_manager.cc:4830] TS cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725): tablet 6b42621fbaf84e8fa42faa2fd37c4b51 (table rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]) successfully deleted
I20240501 14:02:37.829918 6577 raft_consensus.cc:1050] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: attempting to promote NON_VOTER 782e25d5016142c488de06332aa07926 to VOTER
I20240501 14:02:37.830575 6577 consensus_queue.cc:237] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:37.832588 6216 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.832605 5347 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.833113 6576 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.833503 5990 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.836047 5990 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:37.836701 5347 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:37.837615 6216 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:37.841738 5201 catalog_manager.cc:5484] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:37.958884 6676 raft_consensus.cc:1050] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: attempting to promote NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb to VOTER
I20240501 14:02:37.959554 6676 consensus_queue.cc:237] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:37.961783 6084 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.961835 5347 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.962340 6577 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.962684 6676 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.966012 6577 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.966929 5347 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.967087 6084 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:37.971050 5197 catalog_manager.cc:5484] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:37.999505 5990 raft_consensus.cc:1050] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9: attempting to promote NON_VOTER 782e25d5016142c488de06332aa07926 to VOTER
I20240501 14:02:38.000200 5990 consensus_queue.cc:237] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.002383 6216 raft_consensus.cc:1261] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.002475 5347 raft_consensus.cc:1261] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.003036 6576 consensus_queue.cc:1035] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.003367 6676 consensus_queue.cc:1035] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.006274 6676 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.006975 5347 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.007675 6216 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.012117 5201 catalog_manager.cc:5484] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.017910 5742 consensus_queue.cc:237] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.019771 6216 raft_consensus.cc:1261] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:38.020587 5990 consensus_queue.cc:1035] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.023316 6576 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.024088 6216 raft_consensus.cc:2932] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.027117 5190 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 489888747a2941d3bf4c6a7d67cc4a3c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:38.029173 5197 catalog_manager.cc:5484] T 489888747a2941d3bf4c6a7d67cc4a3c P 782e25d5016142c488de06332aa07926 reported cstate change: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.033244 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 489888747a2941d3bf4c6a7d67cc4a3c with delete_type TABLET_DATA_TOMBSTONED (TS cc70ab919f1c47a7a648dfe19791e7e7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55334
I20240501 14:02:38.033674 6622 tablet_replica.cc:330] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: stopping tablet replica
I20240501 14:02:38.033872 6622 raft_consensus.cc:2229] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:38.034144 6622 raft_consensus.cc:2258] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:38.035112 6622 ts_tablet_manager.cc:1875] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:38.040684 6622 ts_tablet_manager.cc:1888] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:38.040822 6622 log.cc:1196] T 489888747a2941d3bf4c6a7d67cc4a3c P cc70ab919f1c47a7a648dfe19791e7e7: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/489888747a2941d3bf4c6a7d67cc4a3c
I20240501 14:02:38.041733 5192 catalog_manager.cc:4830] TS cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725): tablet 489888747a2941d3bf4c6a7d67cc4a3c (table rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]) successfully deleted
I20240501 14:02:38.047044 5742 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33662
I20240501 14:02:38.047333 5742 raft_consensus.cc:595] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.072643 5990 raft_consensus.cc:1050] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9: attempting to promote NON_VOTER 5d61a4d965da4b4f988bd9cb4477f2fb to VOTER
I20240501 14:02:38.073424 5990 consensus_queue.cc:237] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:38.075640 6084 raft_consensus.cc:1261] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 LEARNER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.075776 5347 raft_consensus.cc:1261] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.076261 6576 consensus_queue.cc:1035] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.076551 6676 consensus_queue.cc:1035] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.081411 6676 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:38.086297 6084 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:38.089612 5201 catalog_manager.cc:5484] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.090983 5347 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:38.095924 5742 consensus_queue.cc:237] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:38.097846 6084 raft_consensus.cc:1261] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Refusing update from remote peer d060c465e0fe461da7d8063d9c3847a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:38.098651 5990 consensus_queue.cc:1035] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.101563 5742 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33686
I20240501 14:02:38.101790 5742 raft_consensus.cc:595] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.101452 6676 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:38.102113 6084 raft_consensus.cc:2932] T 4fb6cbe424b54bfa870fd79faa48f815 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:38.105865 5190 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4fb6cbe424b54bfa870fd79faa48f815 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:38.107095 5201 catalog_manager.cc:5484] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 reported cstate change: config changed from index 5 to 6, VOTER cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "d060c465e0fe461da7d8063d9c3847a9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.110889 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb6cbe424b54bfa870fd79faa48f815 with delete_type TABLET_DATA_TOMBSTONED (TS cc70ab919f1c47a7a648dfe19791e7e7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55334
I20240501 14:02:38.111285 6622 tablet_replica.cc:330] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: stopping tablet replica
I20240501 14:02:38.111506 6622 raft_consensus.cc:2229] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:38.111846 6622 raft_consensus.cc:2258] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:38.112776 6622 ts_tablet_manager.cc:1875] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:38.117465 6622 ts_tablet_manager.cc:1888] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:38.117584 6622 log.cc:1196] T 4fb6cbe424b54bfa870fd79faa48f815 P cc70ab919f1c47a7a648dfe19791e7e7: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/4fb6cbe424b54bfa870fd79faa48f815
I20240501 14:02:38.118343 5192 catalog_manager.cc:4830] TS cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725): tablet 4fb6cbe424b54bfa870fd79faa48f815 (table rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]) successfully deleted
I20240501 14:02:38.123553 6609 raft_consensus.cc:1050] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7: attempting to promote NON_VOTER 782e25d5016142c488de06332aa07926 to VOTER
I20240501 14:02:38.124387 6609 consensus_queue.cc:237] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.126579 6216 raft_consensus.cc:1261] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 LEARNER]: Refusing update from remote peer cc70ab919f1c47a7a648dfe19791e7e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.127045 5741 raft_consensus.cc:1261] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Refusing update from remote peer cc70ab919f1c47a7a648dfe19791e7e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:38.127818 6728 consensus_queue.cc:1035] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.128135 6726 consensus_queue.cc:1035] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:38.132068 6726 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.132712 5741 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.136804 5201 catalog_manager.cc:5484] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 reported cstate change: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:38.143628 6216 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 782e25d5016142c488de06332aa07926 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.146533 5347 consensus_queue.cc:237] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.148281 6216 raft_consensus.cc:1261] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Refusing update from remote peer cc70ab919f1c47a7a648dfe19791e7e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:38.148885 6726 consensus_queue.cc:1035] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.151790 6609 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.153441 6216 raft_consensus.cc:2932] T f2f9b497b92144ab9831c75df9a1de8e P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.154613 5192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2f9b497b92144ab9831c75df9a1de8e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:38.156023 5197 catalog_manager.cc:5484] T f2f9b497b92144ab9831c75df9a1de8e P cc70ab919f1c47a7a648dfe19791e7e7 reported cstate change: config changed from index 5 to 6, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.160575 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f9b497b92144ab9831c75df9a1de8e with delete_type TABLET_DATA_TOMBSTONED (TS d060c465e0fe461da7d8063d9c3847a9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52166
I20240501 14:02:38.163480 6741 tablet_replica.cc:330] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: stopping tablet replica
I20240501 14:02:38.163939 6741 raft_consensus.cc:2229] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:38.164243 6741 raft_consensus.cc:2258] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:38.165778 6741 ts_tablet_manager.cc:1875] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:38.171162 6741 ts_tablet_manager.cc:1888] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:38.171334 6741 log.cc:1196] T f2f9b497b92144ab9831c75df9a1de8e P d060c465e0fe461da7d8063d9c3847a9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/f2f9b497b92144ab9831c75df9a1de8e
I20240501 14:02:38.172163 5190 catalog_manager.cc:4830] TS d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007): tablet f2f9b497b92144ab9831c75df9a1de8e (table rebalance_test_table_1 [id=0c7026e7198d41488f0205805fdc2524]) successfully deleted
I20240501 14:02:38.173934 5741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33704
I20240501 14:02:38.174121 5741 raft_consensus.cc:595] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.200251 5741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33720
I20240501 14:02:38.200477 5741 raft_consensus.cc:595] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.216843 5741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33744
I20240501 14:02:38.217079 5741 raft_consensus.cc:595] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.233330 5741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f"
dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33748
I20240501 14:02:38.233546 5741 raft_consensus.cc:595] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:38.292141 6676 raft_consensus.cc:979] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: : Instructing follower 782e25d5016142c488de06332aa07926 to start an election
I20240501 14:02:38.292313 6676 raft_consensus.cc:1067] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Signalling peer 782e25d5016142c488de06332aa07926 to start an election
I20240501 14:02:38.293344 6216 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f"
dest_uuid: "782e25d5016142c488de06332aa07926"
from {username='slave'} at 127.22.5.196:35181
I20240501 14:02:38.293730 6216 raft_consensus.cc:482] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:38.293948 6216 raft_consensus.cc:3037] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.296017 6216 raft_consensus.cc:504] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.297555 6216 leader_election.cc:288] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [CANDIDATE]: Term 2 election: Requested vote from peers d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007), cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725)
I20240501 14:02:38.297997 5741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f" candidate_uuid: "782e25d5016142c488de06332aa07926" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
I20240501 14:02:38.298125 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa72434772e2467580c9eda3fad83c5f" candidate_uuid: "782e25d5016142c488de06332aa07926" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:38.298228 5741 raft_consensus.cc:3032] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:38.298355 5741 raft_consensus.cc:729] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:38.298355 5347 raft_consensus.cc:3037] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.298677 5741 consensus_queue.cc:260] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.299150 5741 raft_consensus.cc:3037] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.301358 5347 raft_consensus.cc:2454] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782e25d5016142c488de06332aa07926 in term 2.
I20240501 14:02:38.301874 5741 raft_consensus.cc:2454] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 782e25d5016142c488de06332aa07926 in term 2.
I20240501 14:02:38.301962 6151 leader_election.cc:302] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782e25d5016142c488de06332aa07926, cc70ab919f1c47a7a648dfe19791e7e7; no voters:
I20240501 14:02:38.302383 6533 raft_consensus.cc:2781] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:38.302765 6533 raft_consensus.cc:686] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 2 LEADER]: Becoming Leader. State: Replica: 782e25d5016142c488de06332aa07926, State: Running, Role: LEADER
I20240501 14:02:38.303207 6533 consensus_queue.cc:237] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.307039 5201 catalog_manager.cc:5484] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 reported cstate change: term changed from 1 to 2, leader changed from d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) to 782e25d5016142c488de06332aa07926 (127.22.5.195). New cstate: current_term: 2 leader_uuid: "782e25d5016142c488de06332aa07926" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.386515 6676 raft_consensus.cc:979] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: : Instructing follower 5d61a4d965da4b4f988bd9cb4477f2fb to start an election
I20240501 14:02:38.386682 6676 raft_consensus.cc:1067] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Signalling peer 5d61a4d965da4b4f988bd9cb4477f2fb to start an election
I20240501 14:02:38.387509 6084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650"
dest_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb"
from {username='slave'} at 127.22.5.196:33219
I20240501 14:02:38.387811 6084 raft_consensus.cc:482] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:38.387995 6084 raft_consensus.cc:3037] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.390123 6084 raft_consensus.cc:504] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:38.391607 6084 leader_election.cc:288] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [CANDIDATE]: Term 2 election: Requested vote from peers cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725), d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007)
I20240501 14:02:38.392274 5741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650" candidate_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d060c465e0fe461da7d8063d9c3847a9"
I20240501 14:02:38.392513 5741 raft_consensus.cc:3032] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:38.392652 5741 raft_consensus.cc:729] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d060c465e0fe461da7d8063d9c3847a9, State: Running, Role: LEADER
I20240501 14:02:38.392974 5741 consensus_queue.cc:260] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:38.393424 5741 raft_consensus.cc:3037] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.395699 5741 raft_consensus.cc:2454] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d61a4d965da4b4f988bd9cb4477f2fb in term 2.
I20240501 14:02:38.396279 6017 leader_election.cc:302] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d61a4d965da4b4f988bd9cb4477f2fb, d060c465e0fe461da7d8063d9c3847a9; no voters:
I20240501 14:02:38.396669 6523 raft_consensus.cc:2781] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:38.397255 6523 raft_consensus.cc:686] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 2 LEADER]: Becoming Leader. State: Replica: 5d61a4d965da4b4f988bd9cb4477f2fb, State: Running, Role: LEADER
I20240501 14:02:38.397833 6523 consensus_queue.cc:237] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:38.398679 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390479ed72084c0eab8e9abc3a23d650" candidate_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cc70ab919f1c47a7a648dfe19791e7e7"
I20240501 14:02:38.398900 5347 raft_consensus.cc:3037] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:38.400926 5347 raft_consensus.cc:2454] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d61a4d965da4b4f988bd9cb4477f2fb in term 2.
I20240501 14:02:38.402040 5201 catalog_manager.cc:5484] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb reported cstate change: term changed from 1 to 2, leader changed from d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) to 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194). New cstate: current_term: 2 leader_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.781631 5741 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Refusing update from remote peer 5d61a4d965da4b4f988bd9cb4477f2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:38.782404 6523 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.786630 5347 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Refusing update from remote peer 5d61a4d965da4b4f988bd9cb4477f2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:38.787458 6523 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.919245 5741 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Refusing update from remote peer 782e25d5016142c488de06332aa07926: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:38.919234 5347 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Refusing update from remote peer 782e25d5016142c488de06332aa07926: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:38.920058 6816 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d060c465e0fe461da7d8063d9c3847a9" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.920500 6815 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:38.937949 6216 consensus_queue.cc:237] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } }
I20240501 14:02:38.939635 5347 raft_consensus.cc:1261] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Refusing update from remote peer 782e25d5016142c488de06332aa07926: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:02:38.940162 6816 consensus_queue.cc:1035] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:38.942508 6815 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.943115 5347 raft_consensus.cc:2932] T aa72434772e2467580c9eda3fad83c5f P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } } }
I20240501 14:02:38.945968 5192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa72434772e2467580c9eda3fad83c5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:38.947180 5197 catalog_manager.cc:5484] T aa72434772e2467580c9eda3fad83c5f P 782e25d5016142c488de06332aa07926 reported cstate change: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New cstate: current_term: 2 leader_uuid: "782e25d5016142c488de06332aa07926" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "782e25d5016142c488de06332aa07926" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 44641 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:38.951716 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet aa72434772e2467580c9eda3fad83c5f with delete_type TABLET_DATA_TOMBSTONED (TS d060c465e0fe461da7d8063d9c3847a9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52166
I20240501 14:02:38.952368 6824 tablet_replica.cc:330] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: stopping tablet replica
I20240501 14:02:38.952556 6824 raft_consensus.cc:2229] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:38.952754 6824 raft_consensus.cc:2258] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:38.953621 6824 ts_tablet_manager.cc:1875] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:38.957957 6824 ts_tablet_manager.cc:1888] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:02:38.958092 6824 log.cc:1196] T aa72434772e2467580c9eda3fad83c5f P d060c465e0fe461da7d8063d9c3847a9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/aa72434772e2467580c9eda3fad83c5f
I20240501 14:02:38.958844 5190 catalog_manager.cc:4830] TS d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007): tablet aa72434772e2467580c9eda3fad83c5f (table rebalance_test_table_2 [id=275618be163d4a0095f5f98e921e0b58]) successfully deleted
I20240501 14:02:39.270898 6084 consensus_queue.cc:237] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } }
I20240501 14:02:39.272766 5347 raft_consensus.cc:1261] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Refusing update from remote peer 5d61a4d965da4b4f988bd9cb4477f2fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:02:39.273460 6840 consensus_queue.cc:1035] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:39.275630 6523 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:39.276190 5347 raft_consensus.cc:2932] T 390479ed72084c0eab8e9abc3a23d650 P cc70ab919f1c47a7a648dfe19791e7e7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } } }
I20240501 14:02:39.278631 5190 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 390479ed72084c0eab8e9abc3a23d650 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:02:39.279987 5201 catalog_manager.cc:5484] T 390479ed72084c0eab8e9abc3a23d650 P 5d61a4d965da4b4f988bd9cb4477f2fb reported cstate change: config changed from index 5 to 7, VOTER d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196) evicted. New cstate: current_term: 2 leader_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cc70ab919f1c47a7a648dfe19791e7e7" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d61a4d965da4b4f988bd9cb4477f2fb" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 45723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:02:39.283577 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet 390479ed72084c0eab8e9abc3a23d650 with delete_type TABLET_DATA_TOMBSTONED (TS d060c465e0fe461da7d8063d9c3847a9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52166
I20240501 14:02:39.283957 6824 tablet_replica.cc:330] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: stopping tablet replica
I20240501 14:02:39.284176 6824 raft_consensus.cc:2229] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:39.284389 6824 raft_consensus.cc:2258] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:39.285240 6824 ts_tablet_manager.cc:1875] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:39.289687 6824 ts_tablet_manager.cc:1888] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:02:39.289809 6824 log.cc:1196] T 390479ed72084c0eab8e9abc3a23d650 P d060c465e0fe461da7d8063d9c3847a9: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/390479ed72084c0eab8e9abc3a23d650
I20240501 14:02:39.290503 5190 catalog_manager.cc:4830] TS d060c465e0fe461da7d8063d9c3847a9 (127.22.5.196:37007): tablet 390479ed72084c0eab8e9abc3a23d650 (table rebalance_test_table_0 [id=4f20dc45841a412d91074b4859913b9c]) successfully deleted
I20240501 14:02:39.424181 6196 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.429800 6064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.434120 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.435074 5722 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:39.607550 6064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.616835 6196 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.617230 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.617928 5722 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:39.758893 6064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.762605 6196 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.764865 5722 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:39.764866 5325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:39.962386 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 5265
I20240501 14:02:39.978816 5387 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:40.106951 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 5265
I20240501 14:02:40.131999 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 6002
W20240501 14:02:40.131948 5677 connection.cc:535] client connection to 127.22.5.193:36725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:40.132218 5677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:40.134491 5677 consensus_peers.cc:485] T 1f179b7c345f4d4eb7f304da148eeabb P d060c465e0fe461da7d8063d9c3847a9 -> Peer cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725): Couldn't send request to peer cc70ab919f1c47a7a648dfe19791e7e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:36725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.138475 6123 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:40.246701 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 6002
W20240501 14:02:40.264590 5677 consensus_peers.cc:485] T 1f0beea9410547eda2c483ece1d6da43 P d060c465e0fe461da7d8063d9c3847a9 -> Peer cc70ab919f1c47a7a648dfe19791e7e7 (127.22.5.193:36725): Couldn't send request to peer cc70ab919f1c47a7a648dfe19791e7e7. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.193:36725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:40.268167 5675 connection.cc:535] client connection to 127.22.5.194:45723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:40.268612 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 6133
W20240501 14:02:40.269243 5675 consensus_peers.cc:485] T 6b42621fbaf84e8fa42faa2fd37c4b51 P d060c465e0fe461da7d8063d9c3847a9 -> Peer 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723): Couldn't send request to peer 5d61a4d965da4b4f988bd9cb4477f2fb. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.274834 6255 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:40.315758 5675 consensus_peers.cc:485] T 4fb6cbe424b54bfa870fd79faa48f815 P d060c465e0fe461da7d8063d9c3847a9 -> Peer 5d61a4d965da4b4f988bd9cb4477f2fb (127.22.5.194:45723): Couldn't send request to peer 5d61a4d965da4b4f988bd9cb4477f2fb. Status: Network error: Client connection negotiation failed: client connection to 127.22.5.194:45723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.384155 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 6133
I20240501 14:02:40.404933 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 5660
I20240501 14:02:40.411298 5782 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:40.542383 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 5660
I20240501 14:02:40.567025 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 5175
I20240501 14:02:40.568159 5236 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:40.703799 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 5175
2024-05-01T14:02:40Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (17399 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240501 14:02:40.798171 22551 test_util.cc:268] Using random seed: -258601525
I20240501 14:02:40.800108 22551 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:02:40.800197 22551 ts_itest-base.cc:113] --------------
I20240501 14:02:40.800251 22551 ts_itest-base.cc:114] 10 tablet servers
I20240501 14:02:40.800323 22551 ts_itest-base.cc:115] 7 replicas per TS
I20240501 14:02:40.800375 22551 ts_itest-base.cc:116] --------------
2024-05-01T14:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:40Z Disabled control of system clock
I20240501 14:02:40.862704 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:41683
--webserver_interface=127.22.5.254
--webserver_port=0
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.5.254:41683
--location_mapping_cmd=/tmp/dist-test-taskA1ASV2/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 14:02:41.235767 6969 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:02:41.236162 6969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:41.236241 6969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:41.236317 6969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:41.244180 6969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:41.244319 6969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:41.244387 6969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:41.244433 6969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:41.244493 6969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:41.255105 6969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskA1ASV2/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.22.5.254:41683
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.5.254:41683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.5.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:41.256494 6969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:41.258387 6969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:41.270326 6974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:41.270839 6975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:41.271318 6977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:41.271986 6969 server_base.cc:1034] running on GCE node
I20240501 14:02:41.272830 6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:41.274426 6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:41.275655 6969 hybrid_clock.cc:648] HybridClock initialized: now 1714572161275592 us; error 43 us; skew 500 ppm
I20240501 14:02:41.276005 6969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:41.278470 6969 webserver.cc:460] Webserver started at http://127.22.5.254:43115/ using document root <none> and password file <none>
I20240501 14:02:41.279254 6969 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:41.279387 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:41.279688 6969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:41.282557 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "50af945f5b5841d091d578cda9d41213"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.283393 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "50af945f5b5841d091d578cda9d41213"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.288997 6969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:02:41.292730 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:41.294353 6969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:41.294553 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
uuid: "50af945f5b5841d091d578cda9d41213"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.294745 6969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:41.313213 6969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:41.314107 6969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:41.314420 6969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:41.332610 6969 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.254:41683
I20240501 14:02:41.332631 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.254:41683 every 8 connection(s)
I20240501 14:02:41.334064 6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:02:41.338037 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:41.340217 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 6969
I20240501 14:02:41.340474 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:02:41.349790 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213: Bootstrap starting.
I20240501 14:02:41.353658 7035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:41.354753 7035 log.cc:824] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:41.357657 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213: No bootstrap required, opened a new log
I20240501 14:02:41.364387 7035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } }
I20240501 14:02:41.364785 7035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:41.364890 7035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50af945f5b5841d091d578cda9d41213, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.365572 7035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } }
I20240501 14:02:41.365789 7035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:41.365917 7035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:41.366073 7035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:41.368203 7035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } }
I20240501 14:02:41.368754 7035 leader_election.cc:302] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50af945f5b5841d091d578cda9d41213; no voters:
I20240501 14:02:41.369436 7035 leader_election.cc:288] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:41.369581 7040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:41.370285 7040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [term 1 LEADER]: Becoming Leader. State: Replica: 50af945f5b5841d091d578cda9d41213, State: Running, Role: LEADER
I20240501 14:02:41.370923 7040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } }
I20240501 14:02:41.371747 7035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:41.374468 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50af945f5b5841d091d578cda9d41213. Latest consensus state: current_term: 1 leader_uuid: "50af945f5b5841d091d578cda9d41213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } } }
I20240501 14:02:41.374886 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:41.375162 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50af945f5b5841d091d578cda9d41213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50af945f5b5841d091d578cda9d41213" member_type: VOTER last_known_addr { host: "127.22.5.254" port: 41683 } } }
I20240501 14:02:41.375579 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:41.378418 7049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:41.383018 7049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:41.392675 7049 catalog_manager.cc:1347] Generated new cluster ID: 76db64a5d8594659a56dd66fe6a62498
I20240501 14:02:41.392803 7049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:41.415346 7049 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:41.416674 7049 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:41.426910 7049 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50af945f5b5841d091d578cda9d41213: Generated new TSK 0
I20240501 14:02:41.427843 7049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:41.438648 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.193:0
--local_ip_for_outbound_sockets=127.22.5.193
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:02:41.801800 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:41.802130 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:41.802273 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:41.809896 7059 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:41.810055 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:41.810184 7059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.193
I20240501 14:02:41.819468 7059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.5.193
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.193
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:41.820915 7059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:41.822757 7059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:41.835539 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:41.836017 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:41.836409 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:41.837062 7059 server_base.cc:1034] running on GCE node
I20240501 14:02:41.837749 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:41.838744 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:41.839955 7059 hybrid_clock.cc:648] HybridClock initialized: now 1714572161839889 us; error 62 us; skew 500 ppm
I20240501 14:02:41.840286 7059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:41.842725 7059 webserver.cc:460] Webserver started at http://127.22.5.193:37055/ using document root <none> and password file <none>
I20240501 14:02:41.843509 7059 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:41.843624 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:41.843945 7059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:41.846813 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0d05240051574799acbbe13e21b262b4"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.847652 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0d05240051574799acbbe13e21b262b4"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.853435 7059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:02:41.857388 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:41.858942 7059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:41.859131 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0d05240051574799acbbe13e21b262b4"
format_stamp: "Formatted at 2024-05-01 14:02:41 on dist-test-slave-5jzj"
I20240501 14:02:41.859355 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:41.885993 7059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:41.886940 7059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:41.887241 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:41.888268 7059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:41.889895 7059 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:41.890002 7059 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:41.890120 7059 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:41.890193 7059 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:41.923360 7059 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.193:41299
I20240501 14:02:41.923403 7185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.193:41299 every 8 connection(s)
I20240501 14:02:41.924754 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:02:41.926218 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7059
I20240501 14:02:41.926446 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:02:41.930207 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.194:0
--local_ip_for_outbound_sockets=127.22.5.194
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:41.939188 7186 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:41.939518 7186 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:41.940388 7186 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:41.993707 7000 ts_manager.cc:194] Registered new tserver with Master: 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:41.996356 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.193:40423
W20240501 14:02:42.291824 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:42.292150 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:42.292248 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:42.299894 7190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:42.300079 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:42.300235 7190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.194
I20240501 14:02:42.309753 7190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.5.194
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.194
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:42.311230 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:42.313733 7190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:42.326424 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.326781 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.327247 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:42.327848 7190 server_base.cc:1034] running on GCE node
I20240501 14:02:42.328505 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:42.329528 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:42.330736 7190 hybrid_clock.cc:648] HybridClock initialized: now 1714572162330689 us; error 44 us; skew 500 ppm
I20240501 14:02:42.331092 7190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:42.333577 7190 webserver.cc:460] Webserver started at http://127.22.5.194:35077/ using document root <none> and password file <none>
I20240501 14:02:42.334374 7190 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:42.334491 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:42.334822 7190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:42.339552 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b3b866abc97f437e8f24eccb653b3dcc"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.340435 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b3b866abc97f437e8f24eccb653b3dcc"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.346530 7190 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240501 14:02:42.350538 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.352088 7190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:02:42.352290 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b3b866abc97f437e8f24eccb653b3dcc"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.352535 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:42.401019 7190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:42.402016 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:42.402326 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:42.403357 7190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:42.404959 7190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:42.405071 7190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.405226 7190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:42.405303 7190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.439234 7190 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.194:41451
I20240501 14:02:42.439289 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.194:41451 every 8 connection(s)
I20240501 14:02:42.440644 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:42.444121 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7190
I20240501 14:02:42.444357 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:02:42.448130 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.195:0
--local_ip_for_outbound_sockets=127.22.5.195
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:42.454941 7318 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:42.455263 7318 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:42.456130 7318 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:42.503311 7000 ts_manager.cc:194] Registered new tserver with Master: b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:02:42.504587 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.194:39767
W20240501 14:02:42.810405 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:42.810708 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:42.810804 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:42.818400 7322 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:42.818529 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:42.818650 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.195
I20240501 14:02:42.827847 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.5.195
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.195
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:42.829209 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:42.830907 7322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:42.843524 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.843988 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.844487 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:42.845099 7322 server_base.cc:1034] running on GCE node
I20240501 14:02:42.845818 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:42.846804 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:42.848027 7322 hybrid_clock.cc:648] HybridClock initialized: now 1714572162847969 us; error 44 us; skew 500 ppm
I20240501 14:02:42.848371 7322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:42.850921 7322 webserver.cc:460] Webserver started at http://127.22.5.195:37997/ using document root <none> and password file <none>
I20240501 14:02:42.851708 7322 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:42.851825 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:42.852154 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:42.854908 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.855729 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.861291 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 14:02:42.864846 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.866273 7322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:42.866439 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
format_stamp: "Formatted at 2024-05-01 14:02:42 on dist-test-slave-5jzj"
I20240501 14:02:42.866637 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:42.892506 7322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:42.893471 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:42.893750 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:42.894809 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:42.896356 7322 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:42.896454 7322 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.896569 7322 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:42.896658 7322 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.930482 7322 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.195:41999
I20240501 14:02:42.930573 7449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.195:41999 every 8 connection(s)
I20240501 14:02:42.931910 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:42.939061 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7322
I20240501 14:02:42.939242 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:02:42.942878 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.196:0
--local_ip_for_outbound_sockets=127.22.5.196
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:42.945294 7450 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:42.945660 7450 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:42.946780 7450 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:43.000381 7186 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
I20240501 14:02:43.015195 7000 ts_manager.cc:194] Registered new tserver with Master: ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:43.016161 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.195:44685
W20240501 14:02:43.321911 7453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:43.322228 7453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:43.322372 7453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:43.330133 7453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:43.330258 7453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:43.330435 7453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.196
I20240501 14:02:43.339735 7453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.5.196
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.196
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:43.341182 7453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:43.342975 7453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:43.355693 7459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:43.356276 7460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:43.356706 7462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:43.357529 7453 server_base.cc:1034] running on GCE node
I20240501 14:02:43.358242 7453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:43.359289 7453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:43.360538 7453 hybrid_clock.cc:648] HybridClock initialized: now 1714572163360472 us; error 47 us; skew 500 ppm
I20240501 14:02:43.360901 7453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:43.363373 7453 webserver.cc:460] Webserver started at http://127.22.5.196:35723/ using document root <none> and password file <none>
I20240501 14:02:43.364250 7453 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:43.364364 7453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:43.364729 7453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:43.367780 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "704fb2d37343406790224dc76b40f210"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.368672 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "704fb2d37343406790224dc76b40f210"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.374703 7453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240501 14:02:43.378526 7468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.379962 7453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:02:43.380126 7453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "704fb2d37343406790224dc76b40f210"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.380331 7453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:43.398115 7453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:43.399094 7453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:43.399367 7453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:43.400494 7453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:43.402238 7453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:43.402324 7453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.402433 7453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:43.402526 7453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.435768 7453 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.196:37547
I20240501 14:02:43.435839 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.196:37547 every 8 connection(s)
I20240501 14:02:43.437165 7453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:43.439150 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7453
I20240501 14:02:43.439353 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:02:43.443560 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.197:0
--local_ip_for_outbound_sockets=127.22.5.197
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:43.452054 7581 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:43.452363 7581 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:43.453263 7581 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:43.502137 7000 ts_manager.cc:194] Registered new tserver with Master: 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547)
I20240501 14:02:43.503486 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.196:39265
I20240501 14:02:43.507417 7318 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:43.802606 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:43.802917 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:43.803050 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:43.810503 7584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:43.810635 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:43.810797 7584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.197
I20240501 14:02:43.819963 7584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.5.197
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.197
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:43.821525 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:43.823380 7584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:43.835840 7590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:43.836292 7591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:43.836731 7593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:43.837467 7584 server_base.cc:1034] running on GCE node
I20240501 14:02:43.838089 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:43.839068 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:43.840274 7584 hybrid_clock.cc:648] HybridClock initialized: now 1714572163840226 us; error 38 us; skew 500 ppm
I20240501 14:02:43.840597 7584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:43.843130 7584 webserver.cc:460] Webserver started at http://127.22.5.197:43751/ using document root <none> and password file <none>
I20240501 14:02:43.843909 7584 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:43.844022 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:43.844314 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:43.847211 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.848045 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.853799 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:02:43.857561 7599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.859148 7584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 14:02:43.859298 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
format_stamp: "Formatted at 2024-05-01 14:02:43 on dist-test-slave-5jzj"
I20240501 14:02:43.859483 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:43.879637 7584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:43.880558 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:43.880800 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:43.881819 7584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:43.883366 7584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:43.883457 7584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.883540 7584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:43.883594 7584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:43.915465 7584 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.197:42421
I20240501 14:02:43.915520 7711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.197:42421 every 8 connection(s)
I20240501 14:02:43.916782 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:02:43.923419 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7584
I20240501 14:02:43.923576 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:02:43.927103 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.198:0
--local_ip_for_outbound_sockets=127.22.5.198
--webserver_interface=127.22.5.198
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:43.931243 7712 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:43.931578 7712 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:43.932629 7712 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:43.980901 7000 ts_manager.cc:194] Registered new tserver with Master: db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:02:43.982184 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.197:60129
I20240501 14:02:44.018922 7450 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:44.279345 7715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:44.279639 7715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:44.279767 7715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:44.287317 7715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:44.287443 7715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:44.287608 7715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.198
I20240501 14:02:44.296823 7715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.5.198
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.198
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:44.298290 7715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:44.300199 7715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:44.316022 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:44.319193 7725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:44.319912 7715 server_base.cc:1034] running on GCE node
W20240501 14:02:44.322736 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:44.323596 7715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:44.324776 7715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:44.326013 7715 hybrid_clock.cc:648] HybridClock initialized: now 1714572164325941 us; error 53 us; skew 500 ppm
I20240501 14:02:44.326406 7715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:44.329154 7715 webserver.cc:460] Webserver started at http://127.22.5.198:44461/ using document root <none> and password file <none>
I20240501 14:02:44.330040 7715 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:44.330215 7715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:44.330570 7715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:44.334273 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c8f737f3f6994db297b47fb23dad1b13"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.335268 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c8f737f3f6994db297b47fb23dad1b13"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.342092 7715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 14:02:44.345917 7730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.347481 7715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:44.347622 7715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c8f737f3f6994db297b47fb23dad1b13"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.347792 7715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:44.364645 7715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:44.365561 7715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:44.365803 7715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:44.366714 7715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:44.368175 7715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:44.368259 7715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.368343 7715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:44.368396 7715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.399670 7715 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.198:45423
I20240501 14:02:44.399730 7842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.198:45423 every 8 connection(s)
I20240501 14:02:44.401080 7715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:02:44.405473 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7715
I20240501 14:02:44.405649 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:02:44.409071 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.199:0
--local_ip_for_outbound_sockets=127.22.5.199
--webserver_interface=127.22.5.199
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:44.415763 7843 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:44.416141 7843 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:44.417289 7843 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:44.477221 7000 ts_manager.cc:194] Registered new tserver with Master: c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:02:44.478127 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.198:33181
I20240501 14:02:44.506458 7581 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:44.767095 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:44.767446 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:44.767596 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:44.775805 7846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:44.775964 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:44.776134 7846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.199
I20240501 14:02:44.785511 7846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.5.199
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.199
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:44.787245 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:44.789113 7846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:44.802115 7853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:44.802752 7852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:44.803326 7855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:44.804064 7846 server_base.cc:1034] running on GCE node
I20240501 14:02:44.804713 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:44.805769 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:44.807024 7846 hybrid_clock.cc:648] HybridClock initialized: now 1714572164806941 us; error 55 us; skew 500 ppm
I20240501 14:02:44.807382 7846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:44.809856 7846 webserver.cc:460] Webserver started at http://127.22.5.199:43513/ using document root <none> and password file <none>
I20240501 14:02:44.810657 7846 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:44.810761 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:44.811101 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:44.813874 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "cc7e7386a9e540eabc4648e147073f1d"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.814678 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "cc7e7386a9e540eabc4648e147073f1d"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.820197 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:02:44.823997 7861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.825546 7846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:44.825721 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cc7e7386a9e540eabc4648e147073f1d"
format_stamp: "Formatted at 2024-05-01 14:02:44 on dist-test-slave-5jzj"
I20240501 14:02:44.825933 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:44.855144 7846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:44.856035 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:44.856302 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:44.857446 7846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:44.859071 7846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:44.859169 7846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.859284 7846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:44.859355 7846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:44.891157 7846 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.199:33993
I20240501 14:02:44.891212 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.199:33993 every 8 connection(s)
I20240501 14:02:44.892536 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:02:44.899633 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7846
I20240501 14:02:44.899803 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:02:44.903347 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.200:0
--local_ip_for_outbound_sockets=127.22.5.200
--webserver_interface=127.22.5.200
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:44.905848 7974 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:44.906184 7974 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:44.907181 7974 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:44.956563 7000 ts_manager.cc:194] Registered new tserver with Master: cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:44.957652 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.199:37529
I20240501 14:02:44.985061 7712 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:45.259544 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:45.259847 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:45.259999 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:45.267596 7978 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:45.267742 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:45.267899 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.200
I20240501 14:02:45.277215 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.5.200
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.200
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:45.278638 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:45.280426 7978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:45.292934 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:45.293362 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:45.293959 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:45.294806 7978 server_base.cc:1034] running on GCE node
I20240501 14:02:45.295526 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:45.296480 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:45.297670 7978 hybrid_clock.cc:648] HybridClock initialized: now 1714572165297621 us; error 40 us; skew 500 ppm
I20240501 14:02:45.297988 7978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:45.300518 7978 webserver.cc:460] Webserver started at http://127.22.5.200:42129/ using document root <none> and password file <none>
I20240501 14:02:45.301317 7978 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:45.301422 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:45.301755 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:45.304319 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "cc3a47f051a546a5a7788d088f061dd5"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.305182 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "cc3a47f051a546a5a7788d088f061dd5"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.310279 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 14:02:45.313619 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.315078 7978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:02:45.315234 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "cc3a47f051a546a5a7788d088f061dd5"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.315433 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:45.338428 7978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:45.339319 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:45.339583 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:45.340574 7978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:45.342098 7978 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:45.342191 7978 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.342299 7978 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:45.342370 7978 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.374300 7978 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.200:43849
I20240501 14:02:45.374370 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.200:43849 every 8 connection(s)
I20240501 14:02:45.375607 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:02:45.376137 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 7978
I20240501 14:02:45.376355 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:02:45.380806 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.201:0
--local_ip_for_outbound_sockets=127.22.5.201
--webserver_interface=127.22.5.201
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:45.391214 8105 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:45.391549 8105 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:45.392365 8105 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:45.441761 7000 ts_manager.cc:194] Registered new tserver with Master: cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849)
I20240501 14:02:45.442693 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.200:58069
I20240501 14:02:45.480901 7843 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:45.736918 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:45.737318 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:45.737428 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:45.744932 8108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:45.745075 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:45.745263 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.201
I20240501 14:02:45.754443 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.22.5.201
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.201
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:45.755759 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:45.757529 8108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:45.770020 8114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:45.770514 8117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:45.771293 8115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:45.772049 8108 server_base.cc:1034] running on GCE node
I20240501 14:02:45.772650 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:45.773700 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:45.774886 8108 hybrid_clock.cc:648] HybridClock initialized: now 1714572165774841 us; error 35 us; skew 500 ppm
I20240501 14:02:45.775207 8108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:45.777503 8108 webserver.cc:460] Webserver started at http://127.22.5.201:39661/ using document root <none> and password file <none>
I20240501 14:02:45.778214 8108 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:45.778316 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:45.778581 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:45.781231 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "41ab226fa6b7403b8b44642e9470517d"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.782006 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "41ab226fa6b7403b8b44642e9470517d"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.787114 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:45.790467 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.791812 8108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:45.791987 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "41ab226fa6b7403b8b44642e9470517d"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:02:45.792179 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:45.823246 8108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:45.824149 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:45.824411 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:45.825500 8108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:45.826967 8108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:45.827062 8108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.827181 8108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:45.827256 8108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:45.858565 8108 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.201:33735
I20240501 14:02:45.858637 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.201:33735 every 8 connection(s)
I20240501 14:02:45.859923 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 14:02:45.862586 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 8108
I20240501 14:02:45.862751 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240501 14:02:45.866633 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.202:0
--local_ip_for_outbound_sockets=127.22.5.202
--webserver_interface=127.22.5.202
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:02:45.875196 8236 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:45.875538 8236 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:45.876534 8236 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:45.923590 7000 ts_manager.cc:194] Registered new tserver with Master: 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735)
I20240501 14:02:45.924580 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.201:38207
I20240501 14:02:45.960374 7974 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:02:46.226104 8239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:46.226416 8239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:46.226562 8239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:46.233831 8239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:02:46.233978 8239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:46.234136 8239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.202
I20240501 14:02:46.242798 8239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.202:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.22.5.202
--webserver_port=0
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.202
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:46.244314 8239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:46.246259 8239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:46.258724 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:46.259302 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:46.259945 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:46.260695 8239 server_base.cc:1034] running on GCE node
I20240501 14:02:46.261336 8239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:46.262259 8239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:46.263507 8239 hybrid_clock.cc:648] HybridClock initialized: now 1714572166263450 us; error 44 us; skew 500 ppm
I20240501 14:02:46.263850 8239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:46.270627 8239 webserver.cc:460] Webserver started at http://127.22.5.202:33341/ using document root <none> and password file <none>
I20240501 14:02:46.271502 8239 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:46.271625 8239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:46.272027 8239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:46.274753 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "1c61c125cd144590ae1e54a2f87d9187"
format_stamp: "Formatted at 2024-05-01 14:02:46 on dist-test-slave-5jzj"
I20240501 14:02:46.275590 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "1c61c125cd144590ae1e54a2f87d9187"
format_stamp: "Formatted at 2024-05-01 14:02:46 on dist-test-slave-5jzj"
I20240501 14:02:46.280911 8239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 14:02:46.284125 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:46.285480 8239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:46.285606 8239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1c61c125cd144590ae1e54a2f87d9187"
format_stamp: "Formatted at 2024-05-01 14:02:46 on dist-test-slave-5jzj"
I20240501 14:02:46.285789 8239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:46.307672 8239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:46.308840 8239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:46.309207 8239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:46.310878 8239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:46.312680 8239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:46.312791 8239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:46.312909 8239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:46.312995 8239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:46.350714 8239 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.202:37399
I20240501 14:02:46.350759 8366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.202:37399 every 8 connection(s)
I20240501 14:02:46.352012 8239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240501 14:02:46.357657 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 8239
I20240501 14:02:46.357879 22551 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240501 14:02:46.365392 8367 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:02:46.365693 8367 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:46.366487 8367 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:02:46.416260 7000 ts_manager.cc:194] Registered new tserver with Master: 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399)
I20240501 14:02:46.417230 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.202:42347
I20240501 14:02:46.426682 22551 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240501 14:02:46.444761 22551 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240501 14:02:46.444918 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 7978
I20240501 14:02:46.445278 8105 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
I20240501 14:02:46.453624 8100 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:46.535414 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 7978
I20240501 14:02:46.551259 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 8108
I20240501 14:02:46.559768 8231 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:46.640403 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 8108
I20240501 14:02:46.656948 22551 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu pid 8239
I20240501 14:02:46.665320 8362 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:46.746719 22551 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu with pid 8239
I20240501 14:02:50.515812 6999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:50.523818 6999 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:50.584383 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.584228 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.584620 7776 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.586131 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.588366 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.590555 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.593433 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.594581 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.597783 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.599507 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.601099 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.601451 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.601958 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.602730 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.603328 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.603734 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.603822 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.602711 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.604710 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.600694 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.605872 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.607363 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.592522 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.617686 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.618763 8400 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:50.622437 8400 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.623747 8400 log.cc:824] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.626912 8400 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:50.627195 8400 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 14:02:50.635887 8400 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.636711 8400 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.636875 8400 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.637874 8400 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.643313 8400 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20240501 14:02:50.643801 8400 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:50.644268 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.645910 8400 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.645959 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.651775 8402 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:50.653215 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.653247 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.653816 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.654052 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.656137 8402 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.657686 8402 log.cc:824] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.667402 8402 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:50.667423 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.669049 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.669900 8399 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:50.670238 8400 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:50.669832 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.670560 8400 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20240501 14:02:50.670231 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.671761 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.672045 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.672164 8400 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.672556 8400 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.672699 8400 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.673137 8400 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
W20240501 14:02:50.673398 7713 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:50.674723 8400 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:50.674772 8399 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.675204 8400 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:50.675344 8402 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20240501 14:02:50.676156 8399 log.cc:824] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.678372 8400 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.686295 8399 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:50.686676 8399 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240501 14:02:50.684882 8405 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:50.692958 8402 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.694567 8400 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:50.694797 8400 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20240501 14:02:50.695593 8400 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.695950 8400 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.696100 8400 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.696447 8400 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.697285 8400 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:50.697619 8402 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.697808 8402 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.698848 8402 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.702807 8399 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.703490 8399 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.703569 8405 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.703619 8399 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.705054 8399 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.708011 8402 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.033s user 0.013s sys 0.007s
I20240501 14:02:50.708508 8402 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:50.708725 8412 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:50.710361 8402 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.711860 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 253a0abd26814fb29f9adede664c5988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.712541 8412 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.713568 7908 tablet_service.cc:1467] Processing CreateTablet for tablet 64b9b1b64def483aaed2a03f7d356915 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.714017 8405 log.cc:824] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.714497 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.715576 8410 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:50.717705 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.719182 8402 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:50.719421 8402 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:02:50.720108 8410 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.720606 8402 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.710805 7909 tablet_service.cc:1467] Processing CreateTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:50.721356 8402 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.721561 8402 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.722147 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:50.721949 8402 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.722514 8412 log.cc:824] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.722910 8402 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:50.723273 8402 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:50.723697 8410 log.cc:824] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.725092 8402 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.730038 8412 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:50.730443 8412 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20240501 14:02:50.731202 8405 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:50.731823 8405 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.007s
I20240501 14:02:50.738940 8402 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:50.739259 8402 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 14:02:50.740535 8402 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.740958 8402 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.741133 8402 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.741490 8402 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.742455 8402 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:50.745985 8412 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.746649 8412 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.746812 8412 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.747787 8412 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.750973 8419 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:50.755136 8405 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.755893 8405 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.756083 8405 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.756798 8412 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.026s user 0.012s sys 0.007s
I20240501 14:02:50.757356 8412 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:50.757172 8405 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.759580 8412 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.765389 8403 raft_consensus.cc:482] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.765668 8403 raft_consensus.cc:504] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.768229 8410 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:50.768630 8410 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.026s
I20240501 14:02:50.771252 8403 leader_election.cc:288] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:50.772579 8405 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.041s user 0.016s sys 0.007s
I20240501 14:02:50.773025 8399 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.086s user 0.017s sys 0.004s
I20240501 14:02:50.773695 8399 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:50.775527 8399 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.777030 8419 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.778513 8419 log.cc:824] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.779181 8405 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:50.781446 8405 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.790226 8403 raft_consensus.cc:482] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.790467 8403 raft_consensus.cc:504] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.791770 8403 leader_election.cc:288] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:50.791677 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:50.792109 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
W20240501 14:02:50.793562 7602 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:50.793867 7602 leader_election.cc:341] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.797215 8405 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:50.797406 8405 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 14:02:50.798511 8405 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.798784 8405 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.798871 8405 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.798746 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:50.799182 8405 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
W20240501 14:02:50.800015 7601 leader_election.cc:341] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.799806 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:50.797708 8410 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
W20240501 14:02:50.800928 7601 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.799626 8412 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:50.801291 8412 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.008s
I20240501 14:02:50.801494 8410 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.802637 8405 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:50.802523 8412 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.802876 8412 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.802819 8410 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.802989 8412 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.804493 8412 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.805331 8412 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:50.805634 8412 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:50.805811 8405 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:50.805682 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
W20240501 14:02:50.806875 7600 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.806864 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
W20240501 14:02:50.807461 7600 leader_election.cc:341] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.808452 8412 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.809058 8410 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.812383 8405 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.813390 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.814471 8419 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:50.814663 8411 raft_consensus.cc:482] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.814782 8419 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.013s
I20240501 14:02:50.814939 8411 raft_consensus.cc:504] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.818349 8399 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:50.818538 8399 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.008s
W20240501 14:02:50.818737 7600 leader_election.cc:341] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.818895 7600 leader_election.cc:302] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: db6b830d9bbe4451bd349c6ad6e14bab; no voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:50.819376 8403 raft_consensus.cc:2726] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:50.819658 8399 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.819979 8399 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.813439 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.820091 8399 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.820650 7404 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.820844 8399 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.821583 8399 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:02:50.821862 8399 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:50.823218 8405 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:50.823411 8405 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20240501 14:02:50.823529 8399 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.825053 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:50.825538 7797 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.825726 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:50.826092 7798 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.826864 8411 leader_election.cc:288] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:50.828613 8410 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.060s user 0.010s sys 0.002s
I20240501 14:02:50.829479 8410 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:50.824173 8405 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.831049 8405 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.831156 8405 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.832233 8410 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.832055 8405 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.836306 8405 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240501 14:02:50.820681 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:50.820817 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:02:50.837764 7601 leader_election.cc:341] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:50.838483 7601 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.838649 7601 leader_election.cc:302] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d
I20240501 14:02:50.839227 8403 raft_consensus.cc:2726] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:50.841413 8410 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:50.841698 8410 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240501 14:02:50.843014 8410 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.843469 8410 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.843611 8410 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.843942 8410 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.844751 8410 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:50.845155 8410 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:50.846869 8410 tablet_bootstrap.cc:654] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.850368 8410 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:50.850535 8410 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:02:50.851150 8410 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.851390 8410 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.851483 8410 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.851752 8410 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.852447 8410 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:50.853057 8399 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:50.853361 8399 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.004s
I20240501 14:02:50.854354 8399 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.854738 8399 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.854882 8399 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.855253 8399 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.856016 8399 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:50.857584 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:50.857980 7273 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:50.861896 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.862143 7404 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:50.863611 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:50.864089 7667 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:50.863999 8419 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.864634 8419 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.864753 8419 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.864905 7731 leader_election.cc:302] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:50.865506 8411 raft_consensus.cc:2781] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:50.865669 8411 raft_consensus.cc:482] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:50.865562 8419 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.865783 8411 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.867074 8419 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.052s user 0.014s sys 0.000s
I20240501 14:02:50.868808 8411 raft_consensus.cc:504] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.869549 8419 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:50.869953 8411 leader_election.cc:288] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:50.870193 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:50.870421 7667 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.870719 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:50.870955 7273 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.871279 8419 tablet_bootstrap.cc:654] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.872354 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
W20240501 14:02:50.873111 7732 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.873358 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
W20240501 14:02:50.873944 7732 leader_election.cc:341] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:50.874069 7273 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:50.874658 7667 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:50.874944 8412 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:50.875146 8412 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.010s
I20240501 14:02:50.875982 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:50.876019 8412 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.876289 8412 raft_consensus.cc:374] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.876363 8412 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.876379 7141 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:50.876703 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:50.876633 8412 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.876881 7140 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.877312 8412 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:02:50.878501 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:50.878726 7404 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.879897 7140 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:50.880319 7733 leader_election.cc:302] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab; no voters: 704fb2d37343406790224dc76b40f210
I20240501 14:02:50.880717 8411 raft_consensus.cc:2781] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:50.882165 8411 raft_consensus.cc:686] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Becoming Leader. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Running, Role: LEADER
I20240501 14:02:50.882412 7404 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:50.882894 8411 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:50.884135 8419 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:50.884367 8419 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240501 14:02:50.884840 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:50.884932 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:50.885229 7928 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:50.885700 8419 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.885957 8419 raft_consensus.cc:374] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.886061 8419 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.886332 8419 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.886945 8419 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:50.888819 8419 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:50.890601 8419 tablet_bootstrap.cc:654] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:50.889842 6999 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198). New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:50.892726 8419 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:50.892858 8419 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:02:50.893503 8419 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.893790 8419 raft_consensus.cc:374] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:50.893884 8419 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.894133 8419 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.894680 8419 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240501 14:02:50.898804 7975 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:50.908177 7844 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:50.933441 7187 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:50.938510 8413 raft_consensus.cc:482] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.938793 8413 raft_consensus.cc:504] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.939299 8446 raft_consensus.cc:482] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 14:02:50.939648 7451 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:50.939538 8446 raft_consensus.cc:504] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.941651 8413 leader_election.cc:288] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:50.942277 8446 leader_election.cc:288] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:50.944677 8403 raft_consensus.cc:482] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.944892 8403 raft_consensus.cc:504] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.946226 8403 leader_election.cc:288] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:02:50.946419 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:50.946655 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:50.946768 7535 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.946851 7798 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.946939 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:50.947151 7140 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.947161 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:50.947453 7928 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
W20240501 14:02:50.947970 7582 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:02:50.949666 7319 tablet.cc:2369] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:50.949735 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.950042 7404 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.950320 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:50.950557 7273 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:50.951033 7601 leader_election.cc:302] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:50.951459 8403 raft_consensus.cc:2781] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:50.951591 8403 raft_consensus.cc:482] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:50.951730 8403 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.952255 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:50.952463 7140 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.952805 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:50.953178 7141 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.958938 8403 raft_consensus.cc:504] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.961405 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:50.961632 7929 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.962081 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:50.962317 7928 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.962317 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:50.962544 7797 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.962270 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:50.962931 7798 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.963274 8403 leader_election.cc:288] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 election: Requested vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:02:50.963694 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:50.963656 7206 leader_election.cc:302] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d; no voters:
I20240501 14:02:50.963910 7141 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.963851 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:50.964095 7535 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.964152 7206 leader_election.cc:302] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d; no voters:
I20240501 14:02:50.964340 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:50.964519 8413 raft_consensus.cc:2781] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:50.964557 7273 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.964648 8413 raft_consensus.cc:482] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:50.964733 8413 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.965792 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:50.966063 7404 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.966915 7141 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.967119 7535 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.967576 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:50.967763 7798 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.967581 8413 raft_consensus.cc:504] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.967749 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:50.968034 7536 raft_consensus.cc:2379] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3b866abc97f437e8f24eccb653b3dcc in current term 1: Already voted for candidate db6b830d9bbe4451bd349c6ad6e14bab in this term.
I20240501 14:02:50.968863 8413 leader_election.cc:288] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:50.969250 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:50.969550 7534 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.970338 7404 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.970513 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:50.970598 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:50.970712 7797 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.970804 7141 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.971711 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:50.971935 7929 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.972250 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:50.972424 7534 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.973420 7797 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.973553 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:50.973754 7928 raft_consensus.cc:3037] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.973810 7141 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.974838 7798 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.975482 7600 leader_election.cc:302] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:50.975914 7534 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.976413 7207 leader_election.cc:302] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13; no voters:
I20240501 14:02:50.976732 8446 raft_consensus.cc:2781] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:50.977833 7929 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.978323 8403 raft_consensus.cc:2781] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:50.978371 7928 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.979251 8446 raft_consensus.cc:686] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Becoming Leader. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Running, Role: LEADER
I20240501 14:02:50.979635 8403 raft_consensus.cc:686] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Becoming Leader. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Running, Role: LEADER
I20240501 14:02:50.980675 8403 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:02:50.981468 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.981727 7404 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.982046 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:50.982379 7405 raft_consensus.cc:2379] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3b866abc97f437e8f24eccb653b3dcc in current term 1: Already voted for candidate db6b830d9bbe4451bd349c6ad6e14bab in this term.
I20240501 14:02:50.982656 7273 raft_consensus.cc:2454] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:50.982935 8413 raft_consensus.cc:2741] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 14:02:50.983155 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:50.983399 7403 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.979928 8446 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:50.986184 7403 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.987031 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:50.987357 7667 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:50.987298 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210980fcec0c4358943ddd2cc7e935a6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:50.988168 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253a0abd26814fb29f9adede664c5988" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:50.988447 7665 raft_consensus.cc:3037] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:50.991354 7665 raft_consensus.cc:2454] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:50.991514 6999 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194). New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:50.991431 6998 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: term changed from 0 to 1, leader changed from <none> to db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197). New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:51.180348 8420 raft_consensus.cc:482] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:51.180581 8420 raft_consensus.cc:504] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:51.182907 8420 leader_election.cc:288] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:51.205166 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:51.205554 7141 raft_consensus.cc:2379] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 704fb2d37343406790224dc76b40f210 in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:02:51.206907 7928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:51.207170 7928 raft_consensus.cc:2454] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 0.
I20240501 14:02:51.208665 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:51.212754 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:51.213114 7665 raft_consensus.cc:2379] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 704fb2d37343406790224dc76b40f210 in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:02:51.214712 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:51.215049 7273 raft_consensus.cc:2379] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 704fb2d37343406790224dc76b40f210 in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:02:51.215597 7469 leader_election.cc:302] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d; no voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:51.216116 8420 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:51.216274 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b9b1b64def483aaed2a03f7d356915" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:51.216481 7403 raft_consensus.cc:2379] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 704fb2d37343406790224dc76b40f210 in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:02:51.219342 8420 raft_consensus.cc:2726] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:51.281069 8411 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.365746 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.372893 8446 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.387578 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.415127 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:51.421506 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:51.421929 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:02:51.423614 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.426000 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.426440 8403 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:51.435499 8456 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:51.446668 8411 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.451027 8411 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.458802 8411 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.481776 8446 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.490571 8446 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.492183 7928 raft_consensus.cc:3037] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:51.506007 8446 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.515630 8487 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.536664 8489 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.569293 8496 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:51.579468 8499 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 455.389 ms
time per row: 0.227695 ms
I20240501 14:02:52.179352 6997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:52.180121 6997 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:52.189553 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.189666 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.189868 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.190033 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.190100 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.190286 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.194625 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.194762 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.195088 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.195124 7120 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.195348 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.195595 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.197976 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.198225 7383 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.198410 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.198642 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.199061 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.199426 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.201916 7646 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.202032 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.202329 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.202241 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.202430 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.202669 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.203794 7776 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.204195 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.205492 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.205880 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.209292 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.209535 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.209764 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.209939 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.210944 7909 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.211369 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.213852 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 9812475b87e9492d8c9450b77d34ea6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.214298 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.218359 7514 tablet_service.cc:1467] Processing CreateTablet for tablet f920ec86006647cc8e4efbf88dabd6cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.218780 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.220796 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.221154 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.221424 8574 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:52.223289 8574 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.225536 8577 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:52.225935 7908 tablet_service.cc:1467] Processing CreateTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:52.226500 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:52.227248 8577 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.228793 8578 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:52.232306 8578 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.232573 8574 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:52.232723 8574 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 14:02:52.233708 8573 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:52.233640 8574 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.233947 8574 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.234035 8574 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.234278 8574 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.235376 8573 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.236232 8577 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:52.236418 8577 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:02:52.237448 8577 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.237812 8577 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.237907 8577 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.238617 8577 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.239449 8577 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:52.241312 8577 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:52.243219 8577 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.244570 8574 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:02:52.244899 8574 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:52.246807 8574 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.248955 8573 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:52.249150 8573 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20240501 14:02:52.249205 8577 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:52.249366 8577 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:02:52.249931 8573 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.250001 8577 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.250213 8573 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.250286 8577 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.250301 8573 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.250365 8577 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.250689 8577 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.251611 8577 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:52.251907 8573 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.252202 8574 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:52.252663 8573 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:52.252935 8573 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:52.253392 8574 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:02:52.254252 8574 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.254565 8574 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.254645 8574 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.254631 8573 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.255580 8574 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.256446 8574 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:52.256841 8574 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:52.259456 8574 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.261329 8581 raft_consensus.cc:482] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.261552 8581 raft_consensus.cc:504] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.268187 8581 leader_election.cc:288] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.276679 8499 raft_consensus.cc:482] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.277019 8499 raft_consensus.cc:504] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.278709 8499 leader_election.cc:288] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.279812 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
W20240501 14:02:52.280746 7206 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.282310 7207 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Not found: Tablet not found: 4e104d44c5fa4a29be94a33cf7e93ba6
W20240501 14:02:52.286402 7206 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Not found: Tablet not found: 4e104d44c5fa4a29be94a33cf7e93ba6
W20240501 14:02:52.286929 7207 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Not found: Tablet not found: 4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:02:52.287060 7207 leader_election.cc:302] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b3b866abc97f437e8f24eccb653b3dcc; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:52.287459 8499 raft_consensus.cc:2726] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:52.377147 8577 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:52.379418 8577 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.390936 8577 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:52.391188 8577 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240501 14:02:52.392238 8577 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.392625 8577 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.392745 8577 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.393158 8577 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.394170 8577 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:52.404598 8591 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:52.407717 8591 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.421866 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
W20240501 14:02:52.422868 7206 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.426607 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:52.437074 8581 raft_consensus.cc:482] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.437368 8581 raft_consensus.cc:504] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.438552 8581 leader_election.cc:288] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:52.441522 8572 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:52.443460 8572 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
W20240501 14:02:52.444453 7208 leader_election.cc:341] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.452044 8575 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:52.454596 8575 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.454847 8574 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:52.455099 8574 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.198s user 0.057s sys 0.025s
I20240501 14:02:52.456182 8574 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.456601 8574 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.456724 8574 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.457063 8574 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.462474 8574 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:52.466881 8499 raft_consensus.cc:482] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.467096 8499 raft_consensus.cc:504] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.468243 8499 leader_election.cc:288] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:52.469472 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:52.470019 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:52.472370 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:02:52.473389 7206 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.473943 7206 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.474239 7207 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.504914 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:52.505247 7667 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:52.506237 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
W20240501 14:02:52.516824 7074 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.517318 7075 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.22.5.254:41683, user_credentials={real_user=slave}} blocked reactor thread for 46920.8us
I20240501 14:02:52.519834 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:52.520085 7141 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:52.521037 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:52.521873 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:02:52.522627 7075 leader_election.cc:341] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.523005 7075 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.470765 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:52.507388 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:52.507624 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:52.526850 8578 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:52.527120 8578 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.298s user 0.052s sys 0.070s
I20240501 14:02:52.528234 8578 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.528616 8578 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.528729 8578 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
W20240501 14:02:52.528537 7206 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.529006 7206 leader_election.cc:302] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc; no voters: c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:52.529050 8578 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.529748 8499 raft_consensus.cc:2726] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:02:52.531687 7074 leader_election.cc:341] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:52.532053 7074 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.534005 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:52.534089 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:52.534125 8578 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 14:02:52.534253 7272 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:52.534350 7273 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:52.534453 8578 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:52.536804 8578 tablet_bootstrap.cc:654] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.457942 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:52.458443 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:52.472041 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
W20240501 14:02:52.540844 7207 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.541447 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:52.541684 7402 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:52.542136 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
W20240501 14:02:52.542675 7074 leader_election.cc:341] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.542708 8578 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:52.542938 8578 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
W20240501 14:02:52.543015 7075 leader_election.cc:341] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.543238 7075 leader_election.cc:302] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, db6b830d9bbe4451bd349c6ad6e14bab; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:52.543761 8581 raft_consensus.cc:2726] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:52.544005 8578 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
W20240501 14:02:52.545104 7075 leader_election.cc:341] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:52.545557 7075 leader_election.cc:302] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:52.545703 8572 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:52.545888 8581 raft_consensus.cc:2726] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:52.545907 8578 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.546021 8578 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.546404 8578 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.546865 8572 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.000s
I20240501 14:02:52.547187 8578 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:02:52.547544 8578 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:52.547580 8573 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:52.548468 8572 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.548990 8573 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.296s user 0.086s sys 0.141s
I20240501 14:02:52.549296 8578 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.549877 8573 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.550185 8573 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.550277 8573 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.550623 8573 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.551352 8573 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:52.551496 8572 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.551668 8572 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.552059 8572 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.552400 8591 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:52.552563 8591 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.148s user 0.007s sys 0.000s
I20240501 14:02:52.553193 8572 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:52.553866 8572 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:52.555284 8591 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.555773 8591 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.555945 8591 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.556126 8573 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:52.557543 8572 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.556329 8591 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.558445 8573 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.559979 8575 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:52.560264 8575 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.000s
I20240501 14:02:52.561447 8575 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.561919 8575 raft_consensus.cc:374] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.562078 8575 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.562508 8575 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.563840 8591 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:02:52.564989 8591 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:52.567790 8591 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.571949 8575 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:02:52.572516 8575 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:52.577229 8575 tablet_bootstrap.cc:654] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.579319 8572 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:52.579499 8572 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.008s
I20240501 14:02:52.579710 8578 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:52.579831 8573 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:52.579886 8578 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.014s
I20240501 14:02:52.580001 8573 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.006s
I20240501 14:02:52.580252 8591 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:52.580255 8572 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.580595 8572 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.580667 8572 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.580505 8591 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240501 14:02:52.580657 8578 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.580749 8573 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.580943 8578 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.581019 8573 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.581035 8578 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.581151 8573 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.580940 8572 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.581568 8572 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:52.581427 8573 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.581858 8572 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:52.582016 8573 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:52.581866 8578 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.582662 8578 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:52.584439 8572 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.585777 8575 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:52.585281 8591 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.585950 8575 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240501 14:02:52.586055 8591 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.586197 8591 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.586942 8575 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.587325 8575 raft_consensus.cc:374] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.587421 8575 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.587702 8591 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.587738 8575 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.588436 8575 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:52.588891 8575 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:52.589457 8591 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:02:52.589768 8591 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:52.591096 8575 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.593230 8591 tablet_bootstrap.cc:654] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:52.593780 8572 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:52.593991 8572 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 14:02:52.594691 8572 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.594879 8572 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.594908 8591 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:52.594952 8572 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.595067 8591 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:02:52.595216 8572 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.595712 8572 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:52.595758 8591 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.595973 8591 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.596071 8591 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.596356 8591 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.596827 8591 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:52.598027 8575 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:52.598209 8575 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240501 14:02:52.599004 8575 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.599265 8575 raft_consensus.cc:374] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:52.599368 8575 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:52.599612 8575 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.600199 8575 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:52.607651 8472 raft_consensus.cc:482] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.607823 8472 raft_consensus.cc:504] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.608839 8472 leader_election.cc:288] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.608943 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:52.608976 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:52.609155 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:52.609225 7533 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.609232 7926 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.609390 7796 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.609764 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:52.609985 7141 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.610044 7600 leader_election.cc:302] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:52.610175 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:52.610343 8472 raft_consensus.cc:2781] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:52.610426 7273 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.610460 8472 raft_consensus.cc:482] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:52.610389 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:52.610553 8472 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.610641 7405 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:02:52.612625 8472 raft_consensus.cc:504] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.613669 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:52.613754 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:52.613909 7533 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.613970 7796 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.614215 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:52.614269 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:52.614395 7141 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.614471 7273 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.616724 7533 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.616803 7796 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.617158 7273 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.617182 7141 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.617280 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:52.617517 7926 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.617660 8472 leader_election.cc:288] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.617861 7600 leader_election.cc:302] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:52.618149 8472 raft_consensus.cc:2781] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:52.618338 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e104d44c5fa4a29be94a33cf7e93ba6" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:52.618517 8472 raft_consensus.cc:686] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Becoming Leader. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Running, Role: LEADER
I20240501 14:02:52.618602 7405 raft_consensus.cc:3037] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.618907 8472 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.620370 7926 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.621322 7405 raft_consensus.cc:2454] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 1.
I20240501 14:02:52.623088 6996 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: term changed from 0 to 1, leader changed from <none> to db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197). New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:52.650494 8489 raft_consensus.cc:482] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.650720 8489 raft_consensus.cc:504] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.651887 8489 leader_election.cc:288] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:52.652129 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:52.652400 7141 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.652423 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:52.652657 7667 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.652776 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:52.652850 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:52.653033 7926 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.653308 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:52.653525 7732 leader_election.cc:302] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:52.653082 7533 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.653563 7405 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.653710 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:52.653859 8489 raft_consensus.cc:2781] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:52.653978 8489 raft_consensus.cc:482] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:52.654033 7273 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:52.654093 8489 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.656121 8489 raft_consensus.cc:504] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.656889 8607 raft_consensus.cc:482] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.657104 8607 raft_consensus.cc:504] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.657356 8489 leader_election.cc:288] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:52.657485 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:52.657694 7141 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.657646 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:52.658143 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:52.658092 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:52.658313 7926 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.658339 7405 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.659883 8607 leader_election.cc:288] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.660475 7141 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.661264 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:52.661329 7926 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.661321 7405 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.661484 7667 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.661839 7533 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.665524 7533 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.665930 7732 leader_election.cc:302] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:52.666277 8489 raft_consensus.cc:2781] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:52.666546 7667 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.667526 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f920ec86006647cc8e4efbf88dabd6cc" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:52.667780 7273 raft_consensus.cc:3037] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.670744 8489 raft_consensus.cc:686] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Becoming Leader. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Running, Role: LEADER
I20240501 14:02:52.671301 7273 raft_consensus.cc:2454] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:02:52.671553 8489 consensus_queue.cc:237] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:52.674242 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:52.674484 7533 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.684103 6996 catalog_manager.cc:5484] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198). New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:52.686403 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:52.686642 7141 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.688026 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:52.688275 7667 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.688258 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:52.688508 7405 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.688851 7862 leader_election.cc:302] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:52.689510 8607 raft_consensus.cc:2781] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:52.689685 8607 raft_consensus.cc:482] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:52.689792 8607 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.689901 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:52.690125 7796 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.692139 8607 raft_consensus.cc:504] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.692675 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:52.692893 7273 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:52.693488 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:52.693701 7667 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.693650 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:52.693835 7141 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.693861 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:52.694054 7273 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.693637 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:52.695120 7533 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.696511 7667 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.696517 7141 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.696671 7273 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.697103 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:52.697455 7796 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.697425 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812475b87e9492d8c9450b77d34ea6b" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:52.697686 7405 raft_consensus.cc:3037] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:52.697961 7862 leader_election.cc:302] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:52.696861 8607 leader_election.cc:288] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:52.698181 7533 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.698463 8607 raft_consensus.cc:2781] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:52.698983 8607 raft_consensus.cc:686] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Becoming Leader. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Running, Role: LEADER
I20240501 14:02:52.699644 8607 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:52.700310 7796 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.700496 7405 raft_consensus.cc:2454] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:52.704216 6996 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7e7386a9e540eabc4648e147073f1d (127.22.5.199). New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:53.052181 8489 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.078240 8472 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:53.089064 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:53.089478 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:53.096835 8627 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:53.104998 8607 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.109158 8625 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.113636 8472 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 14:02:53.149196 8489 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.166267 8621 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.181553 8615 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.184625 8621 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.185163 8621 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.185261 8472 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.193859 8615 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.196491 8661 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.200333 8472 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.212913 8625 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.222733 8625 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.250999 8625 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:53.283272 8625 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:53.468173 7600 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:45423, user_credentials={real_user=slave}} blocked reactor thread for 180155us
W20240501 14:02:53.490814 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 202675us
INSERT report
rows total: 2000
time total: 691.81 ms
time per row: 0.345905 ms
I20240501 14:02:53.949965 6997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:53.950829 6997 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:53.961658 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.961865 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.962158 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.962373 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.963404 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.963850 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.965056 7908 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.965569 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.966248 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.966720 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.969892 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.970136 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.970368 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.970567 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.971110 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.971565 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.973649 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.973837 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.974076 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.974249 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.975096 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.975497 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.977468 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.977608 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.977941 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.977998 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.981633 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.981801 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.982131 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.982240 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.986274 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.986699 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.987532 8730 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:53.989552 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.990002 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.990981 8730 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:53.993556 7909 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.993853 7776 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.993964 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.994287 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.997926 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.998382 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:53.998451 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc517698d5a4ae19525c96334ce7417 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:53.998900 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:54.004885 8736 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:54.007761 8734 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:54.009531 8732 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:54.009929 8736 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.010548 8734 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.012241 8730 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:54.012408 8730 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240501 14:02:54.013263 8730 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.016292 8732 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.016471 8730 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.016595 8730 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.016880 8730 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.024530 8730 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:02:54.024948 8730 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:54.027884 8734 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:54.028914 8732 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:54.029282 8732 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240501 14:02:54.029343 8730 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.029729 8734 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240501 14:02:54.031816 8732 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.031203 8734 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.032591 8732 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.032733 8733 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:54.032735 8732 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.032928 8734 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.033047 8734 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.033311 8732 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.033833 8734 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.034875 8733 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.035494 8732 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:02:54.036715 8736 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:54.037128 8736 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240501 14:02:54.037663 8730 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:54.037851 8730 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240501 14:02:54.038164 8736 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.038528 8736 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.038681 8736 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.038651 8730 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.038942 8730 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.039021 8730 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.039134 8736 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.039278 8730 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.040033 8730 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:54.040334 8730 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:54.040951 8734 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 14:02:54.042026 8732 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:54.042389 8734 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:54.043860 8730 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.047076 8734 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.047392 8732 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.055282 8734 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:54.055665 8731 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:54.056672 8734 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 14:02:54.057080 8737 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:54.057596 8731 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.059170 8737 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.060181 8732 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:54.060410 8732 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20240501 14:02:54.061319 8736 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.024s user 0.000s sys 0.005s
I20240501 14:02:54.061753 8736 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:54.063432 8736 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.059451 8734 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.065266 8734 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.065366 8734 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.065498 8737 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:54.065730 8737 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:02:54.065688 8734 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.066262 8733 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:54.066464 8733 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.012s
I20240501 14:02:54.066747 8732 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.066534 8737 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.067052 8732 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.067107 8737 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.067163 8732 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.067198 8737 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.067449 8737 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.067613 8733 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.068645 8733 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.068866 8734 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:02:54.069202 8734 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:54.069589 8731 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:54.069823 8731 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 14:02:54.070688 8731 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.071072 8731 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.071312 8731 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.071408 8737 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:02:54.071789 8731 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.072055 8732 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.073640 8732 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 14:02:54.073858 8731 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:54.073977 8732 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:54.074285 8731 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:54.074484 8733 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.075181 8737 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:54.075738 8733 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.077220 8737 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.078289 8731 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.078573 8733 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:02:54.079294 8733 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:54.080407 8736 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:54.080639 8736 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240501 14:02:54.081015 8734 tablet_bootstrap.cc:654] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.081889 8736 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.082165 8733 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.082262 8736 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.082343 8736 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.082582 8732 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.082646 8736 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.084223 8736 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:54.084642 8736 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:54.087837 8736 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.088716 8733 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:54.088946 8733 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:02:54.090596 8737 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:54.090797 8737 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240501 14:02:54.090950 8730 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:54.091159 8730 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.011s
I20240501 14:02:54.091743 8731 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:54.091573 8737 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.091861 8737 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.091960 8737 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.092123 8731 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240501 14:02:54.092104 8730 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.092259 8737 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.092453 8730 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.092556 8730 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.092839 8730 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.093142 8731 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.093557 8731 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.093681 8731 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.093775 8730 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:54.094277 8731 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.094782 8737 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:02:54.095090 8737 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:54.095690 8731 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:54.096040 8731 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:54.090476 8733 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.097577 8733 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.097882 8711 raft_consensus.cc:482] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:54.098064 8711 raft_consensus.cc:504] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.099285 8711 leader_election.cc:288] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:02:54.100055 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:54.100235 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:54.100363 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:54.100488 7534 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.100606 7666 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
W20240501 14:02:54.100817 7862 leader_election.cc:341] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:54.100997 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:54.101073 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:54.101281 7798 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.101326 7141 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.101802 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:54.102077 7402 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.102484 8731 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.102596 7862 leader_election.cc:302] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters: b3b866abc97f437e8f24eccb653b3dcc
I20240501 14:02:54.103085 8711 raft_consensus.cc:2781] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:54.103235 8711 raft_consensus.cc:482] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:54.103329 8711 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.103430 8737 tablet_bootstrap.cc:654] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.097724 8733 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.103829 8733 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.104737 8733 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240501 14:02:54.105101 8733 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:54.105623 8734 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:54.105772 8734 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.003s
I20240501 14:02:54.106082 8736 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:54.106240 8736 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:02:54.106359 8711 raft_consensus.cc:504] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.106580 8734 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.106832 8734 raft_consensus.cc:374] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.106932 8733 tablet_bootstrap.cc:654] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:54.106925 8734 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.107206 8734 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.107669 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:54.107898 7141 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.108403 8734 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:54.109917 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:54.110127 7273 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.111440 7141 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.112541 7273 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.113703 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:54.113917 7402 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.107026 8736 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.114910 8736 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.115058 8736 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.115413 8736 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.116328 7402 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.116844 8733 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:54.116976 8732 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:54.117017 8733 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20240501 14:02:54.117262 8732 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.001s
I20240501 14:02:54.117389 8736 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.011s user 0.001s sys 0.003s
I20240501 14:02:54.117877 8733 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.118194 8733 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.118299 8733 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.119037 8732 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.119354 8737 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:54.119387 8732 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.119565 8737 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20240501 14:02:54.119581 8732 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.119824 7862 leader_election.cc:302] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:54.119971 8733 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.120195 8676 raft_consensus.cc:2781] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:54.119995 8732 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.120465 8737 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.120831 8737 raft_consensus.cc:374] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.120935 8737 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.121510 8732 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:54.121268 8737 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.122568 8737 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:54.123838 8711 leader_election.cc:288] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:02:54.124294 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:54.124492 7534 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.126415 8731 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:54.126622 8731 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.002s
I20240501 14:02:54.126839 7534 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.127243 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:54.127430 7666 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.127686 8731 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.128095 8731 raft_consensus.cc:374] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:54.128219 8731 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.128597 8731 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.129365 8731 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:54.129578 7666 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.129961 8733 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240501 14:02:54.129920 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab7d2ddba3440f0a1311f9eda7d3930" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:54.130157 7798 raft_consensus.cc:3037] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.130380 8676 raft_consensus.cc:686] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Becoming Leader. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Running, Role: LEADER
I20240501 14:02:54.131101 8676 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:54.132647 7798 raft_consensus.cc:2454] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.135722 6996 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7e7386a9e540eabc4648e147073f1d (127.22.5.199). New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:54.158138 8676 raft_consensus.cc:482] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:54.158388 8676 raft_consensus.cc:504] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.159765 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:54.159955 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:54.160039 7666 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.160176 7141 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.160199 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:54.160563 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:54.160653 7402 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.160688 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:54.160784 7534 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.160884 7798 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.160877 8676 leader_election.cc:288] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:54.160979 7862 leader_election.cc:302] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:54.161433 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:54.161634 7273 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:54.161823 8676 raft_consensus.cc:2781] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:54.161922 8676 raft_consensus.cc:482] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:54.162067 8676 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.164047 8676 raft_consensus.cc:504] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.165525 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:54.165738 7534 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.165849 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:54.165971 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:54.166071 7666 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.166189 7273 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.168476 7273 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.168695 7534 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.168865 7666 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.168896 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:54.169159 7141 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.169620 8676 leader_election.cc:288] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:54.169930 7862 leader_election.cc:302] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:54.170264 8676 raft_consensus.cc:2781] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:54.170467 8676 raft_consensus.cc:686] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Becoming Leader. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Running, Role: LEADER
I20240501 14:02:54.170553 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:54.170760 7798 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.170961 8676 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.171363 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc517698d5a4ae19525c96334ce7417" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:54.171653 7402 raft_consensus.cc:3037] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.171882 7141 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.174499 7798 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.174909 7402 raft_consensus.cc:2454] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:54.176975 6996 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7e7386a9e540eabc4648e147073f1d (127.22.5.199). New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:54.276145 8597 raft_consensus.cc:482] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:54.276372 8597 raft_consensus.cc:504] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.277602 8597 leader_election.cc:288] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:54.278048 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:54.278154 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:54.278210 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:54.278313 7141 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.278394 7798 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.278445 7534 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.277944 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:54.278820 7666 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.279024 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:54.279189 7207 leader_election.cc:302] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13; no voters:
I20240501 14:02:54.279196 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:54.279371 7402 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.279481 8597 raft_consensus.cc:2781] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:54.279453 7926 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:54.279592 8597 raft_consensus.cc:482] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:54.279695 8597 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.281711 8597 raft_consensus.cc:504] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.282704 8597 leader_election.cc:288] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Requested vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:54.282928 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:54.283074 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:54.283104 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:54.283172 7926 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.283116 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:54.283284 7798 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.283300 7666 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.283336 7534 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.284111 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:54.284322 7141 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.285912 7798 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.286481 7534 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.286480 7926 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.287086 7207 leader_election.cc:302] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d; no voters:
I20240501 14:02:54.287163 7666 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.287716 8597 raft_consensus.cc:2781] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:54.287384 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c89a8bb1e4c499f9619676cc6af97d2" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:54.288615 7402 raft_consensus.cc:3037] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:54.289029 7141 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.289466 8597 raft_consensus.cc:686] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Becoming Leader. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Running, Role: LEADER
I20240501 14:02:54.289897 8597 consensus_queue.cc:237] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:54.291944 7402 raft_consensus.cc:2454] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:54.294095 6996 catalog_manager.cc:5484] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194). New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:54.524487 8676 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.527624 8676 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.551352 8760 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.574100 8607 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.574676 8607 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.575129 8607 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.582556 8711 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.644133 8676 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.685875 8676 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.692054 8760 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.692391 8607 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.710157 8760 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.713611 8597 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:54.728471 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:54.728772 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:02:54.734524 8597 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:54.737929 8765 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:02:54.832598 8597 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.855365 8597 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.868104 8816 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:54.873472 8597 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:55.111725 7864 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 109063us
INSERT report
rows total: 2000
time total: 453.586 ms
time per row: 0.226793 ms
I20240501 14:02:55.346752 6995 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:55.347712 6995 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:55.356760 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.357352 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.357378 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.357591 7909 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.357846 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.358074 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.360344 7120 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.360805 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.362102 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.362530 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.362489 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.362720 7251 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.362963 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.363148 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.363967 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.364382 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.365507 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.365921 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.369582 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.370004 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.370046 7383 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.370609 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.370667 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.371058 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.373275 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.373813 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.376482 7645 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.377395 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.379601 7514 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.380056 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.379979 7907 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.380472 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.384685 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.385102 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.389557 7908 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.390126 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.394878 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.395450 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.403642 8876 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:55.406206 8876 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.408357 8878 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:55.414693 8878 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.417861 8876 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:55.418133 8876 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 14:02:55.419538 8876 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.419979 8876 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.420172 8876 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.420574 8876 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.422328 8876 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:55.422672 8876 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:55.425025 8876 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.425601 8877 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:55.426151 8881 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:55.426528 8882 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:55.428097 8877 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.428896 8882 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.432044 8878 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:55.432291 8878 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.007s
I20240501 14:02:55.432693 8876 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:55.432727 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 2264386e494644aa84e8109c816aaefa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.432871 8876 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 14:02:55.432958 7777 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f7d11351b4f8d84021720488abf25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:55.433192 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.433357 8878 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.433641 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:55.433638 8878 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.433724 8878 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.433622 8876 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.433905 8876 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.433974 8876 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.434010 8878 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.439903 8878 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 14:02:55.441671 8881 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.446236 8878 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:55.449087 8878 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.449553 8875 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:55.453595 8876 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.454622 8878 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:55.454872 8878 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 14:02:55.455819 8882 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:55.455819 8878 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.456131 8878 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.456207 8878 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.456485 8878 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.457414 8876 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.024s user 0.002s sys 0.001s
I20240501 14:02:55.457942 8873 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:55.458308 8875 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.458514 8882 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240501 14:02:55.459481 8876 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:55.459990 8873 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.460415 8882 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.460758 8882 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.460880 8882 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.461315 8881 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:55.461478 8881 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20240501 14:02:55.461527 8878 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:02:55.462363 8877 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:55.462538 8877 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20240501 14:02:55.462327 8881 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.462633 8881 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.462731 8881 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.462956 8881 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.463939 8881 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:55.465335 8876 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.465968 8881 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:55.466351 8882 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.467072 8882 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:02:55.467412 8882 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:55.468429 8881 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.468436 8877 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.468756 8873 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:55.468767 8877 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.468895 8877 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.469061 8873 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 14:02:55.469168 8877 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.469874 8873 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.470237 8873 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.470381 8873 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.470664 8873 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.472002 8875 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:55.472189 8875 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:02:55.472726 8597 raft_consensus.cc:482] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.472931 8597 raft_consensus.cc:504] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.473155 8875 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.473588 8875 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.473742 8875 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.473796 8882 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.474356 8597 leader_election.cc:288] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:55.474124 8875 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.475276 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:55.475554 7405 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:55.475994 8875 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:55.476085 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:55.476294 7798 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:55.477154 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:55.477635 8875 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
W20240501 14:02:55.478384 7208 leader_election.cc:341] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.478459 8873 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240501 14:02:55.479341 8873 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:55.479321 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:55.480088 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:02:55.480792 7207 leader_election.cc:341] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:55.480969 7207 leader_election.cc:341] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.481155 8873 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.481567 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:55.482167 8875 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
W20240501 14:02:55.483762 7206 leader_election.cc:341] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.483932 7206 leader_election.cc:302] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:55.487442 8877 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240501 14:02:55.487556 8597 raft_consensus.cc:2726] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:55.489454 8877 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:55.491170 8877 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.491941 8873 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:55.492177 8873 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240501 14:02:55.493043 8873 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.493427 8873 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.493528 8873 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.493886 8873 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.495158 8873 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:55.495488 8873 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:55.495514 8875 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:55.495689 8875 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240501 14:02:55.496452 8875 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.496768 8875 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.496855 8875 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.497922 8881 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:55.498099 8881 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240501 14:02:55.498816 8881 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.499133 8881 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.499259 8873 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.499312 8881 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.499644 8881 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.500416 8892 raft_consensus.cc:482] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.500648 8892 raft_consensus.cc:504] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.501240 8881 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:55.501535 8881 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:55.502478 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
W20240501 14:02:55.503157 7074 leader_election.cc:341] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.503197 8875 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.504112 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
W20240501 14:02:55.504591 7074 leader_election.cc:341] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.504654 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:55.504910 7535 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.505172 8881 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.505419 8875 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:02:55.506084 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:55.506237 8877 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:55.506054 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:55.506319 7798 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.506953 8875 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:55.507556 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:55.507802 7667 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.508155 8877 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 14:02:55.508970 8877 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.509318 8877 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.509413 8877 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.509688 8877 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.506249 8892 leader_election.cc:288] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
W20240501 14:02:55.506651 7075 leader_election.cc:341] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:55.510737 8875 tablet_bootstrap.cc:654] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.510746 7074 leader_election.cc:302] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab; no voters: b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:55.513494 8892 raft_consensus.cc:2781] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:55.513904 8892 raft_consensus.cc:482] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:55.514212 8892 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.516587 8875 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:55.516728 8875 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:02:55.517467 8875 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.517725 8875 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.517805 8875 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.519487 8873 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:55.519687 8873 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240501 14:02:55.520397 8873 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.520714 8873 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.520821 8873 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.521106 8873 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.522095 8878 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:55.522899 8873 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:55.523459 8875 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.525020 8892 raft_consensus.cc:504] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.525869 8875 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:02:55.528796 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:55.529088 7535 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.529717 8878 tablet_bootstrap.cc:654] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.532184 7535 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.533231 8881 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:55.533239 8877 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:02:55.533428 8881 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.002s
I20240501 14:02:55.534294 8881 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.534600 8881 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.534715 8881 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.535020 8881 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.535784 8881 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:55.535936 8877 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:55.537585 8882 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:55.537669 8877 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.537837 8876 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:55.537950 8882 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.010s
I20240501 14:02:55.538012 8876 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.004s
I20240501 14:02:55.539513 8876 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.540226 8876 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.540520 8876 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.541209 8876 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.540068 8882 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.542276 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:55.542647 8876 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:02:55.542863 7405 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.543251 8882 raft_consensus.cc:374] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.543385 8882 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.543738 8878 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:55.543891 8878 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:02:55.543753 8882 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.544509 8878 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.544688 8878 raft_consensus.cc:374] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.544744 8878 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.544966 8878 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.540880 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:55.545904 7272 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.547770 8882 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240501 14:02:55.548578 7272 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.549542 7405 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.549837 8882 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:55.550233 7074 leader_election.cc:302] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:55.550616 8901 raft_consensus.cc:2781] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:55.550848 8901 raft_consensus.cc:686] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Running, Role: LEADER
I20240501 14:02:55.551771 8882 tablet_bootstrap.cc:654] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:55.551520 8901 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:55.555368 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:55.555614 7926 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.558432 7926 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.558691 8882 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:55.558853 8882 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240501 14:02:55.559571 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:55.559782 7798 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.559780 8882 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.560037 8882 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.560132 8882 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.545631 8878 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:55.561403 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:55.561646 7667 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.562566 7798 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.560518 8882 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.563510 8892 leader_election.cc:288] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Requested vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:55.564167 7667 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.564390 8882 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:02:55.566080 6998 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d05240051574799acbbe13e21b262b4 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:55.574275 8877 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:55.574479 8877 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240501 14:02:55.575129 8877 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.575321 8877 raft_consensus.cc:374] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:55.575402 8877 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:55.575691 8877 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.576218 8877 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:55.644929 8843 raft_consensus.cc:482] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.645156 8843 raft_consensus.cc:504] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.646312 8843 leader_election.cc:288] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:55.646363 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:55.646441 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:55.646626 7667 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.646647 7535 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.646735 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:55.646996 7798 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.646979 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:55.647226 7139 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.647506 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:55.647647 7864 leader_election.cc:302] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:55.647718 7405 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.647698 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:55.647915 7272 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:02:55.647948 8843 raft_consensus.cc:2781] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:55.648061 8843 raft_consensus.cc:482] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:55.648155 8843 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.650246 8843 raft_consensus.cc:504] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.651284 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:55.651364 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:55.651460 7667 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.651563 7535 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.651559 8843 leader_election.cc:288] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Requested vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:55.651700 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:55.651903 7139 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.652446 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:55.652679 7798 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.654429 7535 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.654817 7139 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.655110 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:55.655359 7405 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.655413 7798 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.655498 7667 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.656020 7862 leader_election.cc:302] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:55.656155 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f7d11351b4f8d84021720488abf25" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:55.656349 8843 raft_consensus.cc:2781] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:55.656370 7272 raft_consensus.cc:3037] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.656430 8901 raft_consensus.cc:482] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.656610 8901 raft_consensus.cc:504] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.656996 8843 raft_consensus.cc:686] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Becoming Leader. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Running, Role: LEADER
I20240501 14:02:55.657451 8843 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.657750 8901 leader_election.cc:288] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:55.658560 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:55.658815 7536 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.659252 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:55.659379 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:55.659492 7273 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.659588 7926 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.659469 7272 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.658145 7405 raft_consensus.cc:2454] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:02:55.659963 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:55.660208 7402 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.660254 7074 leader_election.cc:302] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d; no voters:
I20240501 14:02:55.660492 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:55.660569 8901 raft_consensus.cc:2781] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:55.660667 8901 raft_consensus.cc:482] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:55.660687 7798 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.660777 8901 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.662366 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:55.662647 7667 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:55.663269 8901 raft_consensus.cc:504] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.663650 6998 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7e7386a9e540eabc4648e147073f1d (127.22.5.199). New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:55.664717 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:55.664925 7667 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.664862 8901 leader_election.cc:288] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Requested vote from peers db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:55.665083 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:55.665426 7536 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.665519 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:55.665714 7926 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.667563 7667 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.668177 7536 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.668262 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:55.668450 7926 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.668484 7797 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.668790 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:55.669003 7272 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.669109 7074 leader_election.cc:302] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:55.669395 8901 raft_consensus.cc:2781] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:55.669533 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da808a7f774e5ea7bd7ef298f1180e" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:55.669770 7402 raft_consensus.cc:3037] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:55.669962 8901 raft_consensus.cc:686] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Running, Role: LEADER
I20240501 14:02:55.670300 8901 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:55.671550 7797 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.672734 7402 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.672849 7272 raft_consensus.cc:2454] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:55.674943 6998 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d05240051574799acbbe13e21b262b4 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:55.956341 8892 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:55.958464 8901 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:55.988155 8892 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.003484 8902 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.008666 8902 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.011191 8901 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.039630 8843 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:56.100940 8911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:56.101333 8911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:56.109004 8911 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:56.115720 8902 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.120114 8843 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.127986 8902 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.131104 8902 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.138649 8892 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.138912 8902 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 14:02:56.146855 8843 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.163623 8892 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.179651 8843 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.182772 8843 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:56.184959 8951 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:56.461472 7074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:41999, user_credentials={real_user=slave}} blocked reactor thread for 122696us
INSERT report
rows total: 2000
time total: 509.323 ms
time per row: 0.254662 ms
I20240501 14:02:56.831604 6998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:56.832392 6998 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:56.841322 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.841845 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.842062 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.842274 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.842471 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.842769 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.844058 7120 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.844470 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.846479 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.846910 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.849287 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.849578 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.849637 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.849782 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.850028 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.850039 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.854560 9015 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:56.856353 9015 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.856320 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.856642 7384 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.856868 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.857098 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.865368 9015 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:56.865634 9015 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 14:02:56.866539 9015 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.866806 7515 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.866966 9015 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.867089 9015 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.867441 9015 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.868206 9015 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:56.869010 7646 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.869545 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.870152 9014 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:56.871889 9014 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.872432 7778 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.872900 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.874361 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.876179 7907 tablet_service.cc:1467] Processing CreateTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.876576 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.880960 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.881420 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.889631 9015 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:56.889847 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.890394 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.893782 9015 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.894109 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.894564 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.899797 9015 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:56.903025 9015 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:02:56.904518 9015 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.904865 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.905155 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.905544 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.905694 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.907232 9015 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.907321 9015 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.908159 9015 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.911023 9015 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:02:56.912357 9015 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:56.915413 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.915876 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.917038 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 5efbd81b803841a288eed30cb91dad31 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:56.917479 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:56.917578 9022 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:56.919349 9022 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.924583 9014 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:56.925519 9014 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.015s
I20240501 14:02:56.927353 9014 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.928198 9014 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.928474 9014 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.928891 9014 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.930168 9014 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:56.931125 9014 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:56.934881 9014 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.941181 9015 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.955391 9015 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:56.959491 9015 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.003s
I20240501 14:02:56.961642 9020 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:56.964110 9020 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.966053 9016 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:56.967773 9015 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.968094 9015 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.968206 9015 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.968818 9016 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.969748 9022 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:56.969936 9022 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.015s
I20240501 14:02:56.970706 9022 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.970988 9022 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.971074 9022 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.971334 9022 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:56.971944 9022 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:56.972250 9022 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:56.975143 9019 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:56.974980 9015 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.975879 9022 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.977833 9015 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240501 14:02:56.985653 9022 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:56.985874 9022 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240501 14:02:56.986256 9021 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:56.986655 9022 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.986970 9022 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:56.987048 9022 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:56.987303 9022 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:56.989065 9021 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.992678 9019 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:56.993631 9022 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:02:56.994004 9022 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:56.995697 9022 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.005431 9014 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:57.005648 9014 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.031s
I20240501 14:02:57.006505 9014 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.006799 9014 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.006891 9014 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.007194 9014 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.007844 9014 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:57.008244 9014 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:57.009934 9014 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.014918 9022 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:57.015216 9022 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240501 14:02:57.015866 9022 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.016125 9022 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.016227 9022 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.016487 9022 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.017051 9022 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:57.017500 8597 raft_consensus.cc:482] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.017635 8597 raft_consensus.cc:504] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.017916 9020 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:57.018182 9020 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240501 14:02:57.018711 8597 leader_election.cc:288] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.019410 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:57.019241 9020 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.019623 7139 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.019640 9020 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.019744 9020 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.022378 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
W20240501 14:02:57.023146 7206 leader_election.cc:341] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.023699 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:57.024557 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
W20240501 14:02:57.026718 7206 leader_election.cc:341] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:57.026935 7206 leader_election.cc:341] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.027164 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:57.028339 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:02:57.029143 7207 leader_election.cc:341] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.029306 7207 leader_election.cc:302] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:57.030040 8597 raft_consensus.cc:2726] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:57.030570 9016 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:57.030797 9016 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
W20240501 14:02:57.031210 7207 leader_election.cc:341] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.032070 9016 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.032413 9016 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.032614 9021 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:57.032552 9016 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.032924 9020 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.032999 9016 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.033417 9019 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:57.033687 9021 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.003s
I20240501 14:02:57.033926 9019 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240501 14:02:57.035030 9021 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.036706 9019 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.042506 9016 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20240501 14:02:57.043087 9019 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.043620 9020 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240501 14:02:57.043793 9019 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.044479 9020 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:57.044483 9016 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:57.044595 9019 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.047703 9016 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.048167 9019 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:02:57.048342 9021 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.048516 9021 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.048890 9019 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:57.049240 9021 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.051435 9021 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:02:57.051898 9020 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.052515 9014 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:57.052217 9036 raft_consensus.cc:482] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.052712 9014 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.022s
I20240501 14:02:57.052750 9036 raft_consensus.cc:504] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.053633 9014 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.054075 9014 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.054114 9036 leader_election.cc:288] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.054188 9014 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.054198 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:57.054458 7139 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 0.
I20240501 14:02:57.054473 9014 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.055094 9014 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:57.055070 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:57.055298 7667 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 0.
I20240501 14:02:57.055583 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:57.055565 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:57.055624 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:57.055761 7929 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 0.
I20240501 14:02:57.055817 7271 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 0.
W20240501 14:02:57.056453 7469 leader_election.cc:341] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.052574 9021 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:57.057008 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:57.057300 7469 leader_election.cc:302] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters: ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:57.057617 9036 raft_consensus.cc:2781] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.057731 9036 raft_consensus.cc:482] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.057814 9036 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.058950 9019 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.061213 9036 raft_consensus.cc:504] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
W20240501 14:02:57.061671 7469 leader_election.cc:341] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.061898 9021 tablet_bootstrap.cc:654] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.064265 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:57.064517 7139 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.067677 7139 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 1.
I20240501 14:02:57.069370 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:57.069572 7271 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.070006 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
W20240501 14:02:57.070717 7469 leader_election.cc:341] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.072156 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:57.072435 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:57.072667 7667 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:02:57.072921 7469 leader_election.cc:341] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:57.073748 9036 leader_election.cc:288] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.075655 7667 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 1.
I20240501 14:02:57.076392 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "704fb2d37343406790224dc76b40f210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:57.076610 7929 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.077093 7271 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 1.
I20240501 14:02:57.078424 9019 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:57.078601 9019 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20240501 14:02:57.078882 7469 leader_election.cc:302] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, db6b830d9bbe4451bd349c6ad6e14bab; no voters: c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:02:57.079097 9016 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:57.079285 9016 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20240501 14:02:57.079298 7929 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704fb2d37343406790224dc76b40f210 in term 1.
I20240501 14:02:57.079326 9036 raft_consensus.cc:2781] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.079581 9020 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:57.079771 9020 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20240501 14:02:57.080590 9016 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.081009 9016 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.080808 9019 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.081096 9016 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.081144 9019 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.081236 9019 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.081470 9016 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.081935 9036 raft_consensus.cc:686] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Becoming Leader. State: Replica: 704fb2d37343406790224dc76b40f210, State: Running, Role: LEADER
I20240501 14:02:57.082207 9016 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:57.082212 9020 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.082670 9016 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:57.082628 9019 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.083289 9019 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:02:57.083612 9019 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:57.084029 9036 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.084312 9016 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.086952 9020 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.087077 9020 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.087504 9020 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.095403 9019 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.095403 9021 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:57.095621 9021 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240501 14:02:57.096484 9021 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.096907 9021 raft_consensus.cc:374] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.096998 9021 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.097318 9021 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.097966 9021 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:57.097698 9020 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20240501 14:02:57.098368 9021 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:57.099681 9020 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:57.101421 9020 tablet_bootstrap.cc:654] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.104772 9021 tablet_bootstrap.cc:654] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:57.105865 9020 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:57.107787 9020 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:02:57.108776 9016 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:57.108983 9016 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.010s
I20240501 14:02:57.109164 6991 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 704fb2d37343406790224dc76b40f210 (127.22.5.196). New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:57.110184 9016 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.110471 9016 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.110596 9020 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.110929 9020 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.111078 9020 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.111471 9016 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.111377 9020 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.111807 9016 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.112272 9020 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:57.112905 9019 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:57.113090 9019 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.007s
I20240501 14:02:57.113176 9016 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:57.114331 9021 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:57.114372 9019 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.114630 9019 raft_consensus.cc:374] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.114750 9019 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.115038 9019 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.115612 9021 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240501 14:02:57.115752 9019 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:57.116430 9021 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.116721 9021 raft_consensus.cc:374] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:57.116828 9021 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.117094 9021 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.117781 9021 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:57.130103 9035 raft_consensus.cc:482] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.130323 9035 raft_consensus.cc:504] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.132531 9035 leader_election.cc:288] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.192701 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:57.193058 7271 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:57.192865 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:57.193307 7536 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:57.194785 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:57.195050 7139 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:57.195537 7340 leader_election.cc:302] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:57.195911 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:57.196111 9035 raft_consensus.cc:2781] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.196151 7929 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:57.196246 9035 raft_consensus.cc:482] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.196346 9035 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.199262 9035 raft_consensus.cc:504] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.200584 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:57.200814 7139 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.200996 9035 leader_election.cc:288] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.201704 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:57.201912 7271 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.202800 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:57.203039 7536 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.203593 7139 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.204547 7271 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.205336 7536 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.205554 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:57.205788 7797 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:57.205843 7339 leader_election.cc:302] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:57.206161 9035 raft_consensus.cc:2781] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.206135 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:57.206386 7929 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.206483 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:57.206450 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:57.206674 7666 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.208446 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efbd81b803841a288eed30cb91dad31" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:57.208679 7797 raft_consensus.cc:3037] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.209501 7929 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.209815 7666 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.211791 7797 raft_consensus.cc:2454] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:57.212051 9035 raft_consensus.cc:686] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Running, Role: LEADER
I20240501 14:02:57.212713 9035 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.217855 6998 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:57.272101 9030 raft_consensus.cc:482] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.272289 9030 raft_consensus.cc:504] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.273442 9030 leader_election.cc:288] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:57.273545 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:57.273733 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:57.273766 7797 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.273947 7666 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.274037 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:57.274255 7139 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.274338 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:57.274580 7402 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.274859 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:57.274971 7206 leader_election.cc:302] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:02:57.275070 7536 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.275243 9030 raft_consensus.cc:2781] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:57.275370 9030 raft_consensus.cc:482] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:57.275472 9030 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.275483 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:57.275698 7929 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 0.
I20240501 14:02:57.278031 9030 raft_consensus.cc:504] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.279011 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:57.279106 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:57.279171 7797 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.279320 7666 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.279364 9030 leader_election.cc:288] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:57.279772 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:57.279959 7139 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.280135 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:57.280350 7402 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.281914 7797 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.282011 7666 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.282516 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:57.282624 7139 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.282727 7929 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.282940 7402 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.283367 7206 leader_election.cc:302] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:57.283784 9030 raft_consensus.cc:2781] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:57.283984 9030 raft_consensus.cc:686] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Becoming Leader. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Running, Role: LEADER
I20240501 14:02:57.284482 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab9f466bd0244debff3e228ba99eb51" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:57.284411 9030 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:57.284714 7536 raft_consensus.cc:3037] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.287209 7536 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.288081 6998 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194). New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:57.289057 7929 raft_consensus.cc:2454] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 1.
I20240501 14:02:57.372483 8925 raft_consensus.cc:482] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:57.372717 8925 raft_consensus.cc:504] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:02:57.373925 8925 leader_election.cc:288] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:02:57.374060 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:57.374382 7139 raft_consensus.cc:2379] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8f737f3f6994db297b47fb23dad1b13 in current term 1: Already voted for candidate 704fb2d37343406790224dc76b40f210 in this term.
I20240501 14:02:57.374557 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:57.374624 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:57.374825 7402 raft_consensus.cc:2454] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:02:57.375358 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:57.375533 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:57.375643 7271 raft_consensus.cc:2379] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8f737f3f6994db297b47fb23dad1b13 in current term 1: Already voted for candidate 704fb2d37343406790224dc76b40f210 in this term.
I20240501 14:02:57.375854 7666 raft_consensus.cc:2379] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8f737f3f6994db297b47fb23dad1b13 in current term 1: Already voted for candidate 704fb2d37343406790224dc76b40f210 in this term.
I20240501 14:02:57.375775 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56b1674c3ec4bdc9c2bd82a231d0112" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:57.376140 7929 raft_consensus.cc:2379] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8f737f3f6994db297b47fb23dad1b13 in current term 1: Already voted for candidate 704fb2d37343406790224dc76b40f210 in this term.
I20240501 14:02:57.376266 7731 leader_election.cc:302] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:02:57.376593 8925 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.378564 8925 raft_consensus.cc:2726] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:57.462855 7402 raft_consensus.cc:3037] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:57.468134 9041 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.477593 9041 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.542580 9041 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.548439 9045 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.552491 9041 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.564738 9045 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.595045 9035 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.598964 9060 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.617013 9060 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.623029 9060 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.624711 9060 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.625710 9060 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.751705 8597 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:57.773547 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:57.773883 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:57.782063 9063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:57.789616 8597 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.816716 8597 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 14:02:57.819926 8597 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.828967 9102 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:57.837844 9102 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 478.722 ms
time per row: 0.239361 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:02:59.459426 6993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:02:59.460449 6993 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:02:59.470003 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.470543 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.471073 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.471235 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.471608 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.471642 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.473582 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.474016 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.477586 7777 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.478024 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.479127 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.479525 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.480693 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.481367 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.481866 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 60cf39e6be52426c93c1534754be7264 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.482282 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.492405 9171 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:59.494089 9171 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.496050 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.496512 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.496817 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.497318 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.499210 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.499657 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.500684 7909 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.500975 7645 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.501197 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.501382 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.505586 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.506039 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.510574 7514 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.511057 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.513619 7776 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.514194 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.515552 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.515993 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.516846 7908 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.517366 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.520485 9172 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:59.522249 7253 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.522773 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.523344 9173 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:59.524224 7384 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.524782 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.525015 9173 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.526182 9177 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:59.527884 9177 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.529330 9172 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.536404 9175 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:59.536842 9173 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:59.537084 9173 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 14:02:59.537632 9172 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:59.538033 9173 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.538331 9173 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.538462 9173 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.538604 9172 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20240501 14:02:59.538712 9175 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.538731 9173 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.540613 9173 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:59.540946 9173 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:59.542500 9173 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.542524 9171 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:59.543247 9171 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240501 14:02:59.544976 9171 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.544844 9172 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.545715 9172 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.546229 9172 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.548007 9171 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.548305 9171 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.548959 9171 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.549821 9171 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:02:59.551492 9171 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:59.551692 9172 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.554966 9171 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.556434 9173 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:59.556623 9173 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240501 14:02:59.557451 9173 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.557763 9173 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.557844 9173 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.558141 9173 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.554046 7119 tablet_service.cc:1467] Processing CreateTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff (DEFAULT_TABLE table=rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:59.559212 9172 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240501 14:02:59.559674 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:59.559693 9170 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:59.561302 9173 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:59.563066 9172 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:59.564605 9173 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:02:59.566735 9170 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.567281 9173 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.568414 9172 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.574422 9176 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:59.577575 9176 tablet_bootstrap.cc:654] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.577697 9172 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:59.577937 9172 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240501 14:02:59.578881 9172 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.579269 9172 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.579361 9172 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.579563 9170 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:59.579771 9170 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.004s
I20240501 14:02:59.580740 9170 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.581172 9170 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.581283 9170 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.582512 9177 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:59.582764 9177 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20240501 14:02:59.583829 9177 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.584426 9177 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.584628 9177 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.585031 9177 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.586210 9177 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:59.589313 9177 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:59.589329 9172 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.590924 9172 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 14:02:59.591256 9172 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:02:59.593847 9172 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.594089 9170 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.595075 9170 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:02:59.595398 9177 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.596743 9175 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:59.596954 9175 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.011s
I20240501 14:02:59.598047 9170 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:59.601562 9175 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.602007 9175 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.602128 9175 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.602468 9175 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.603668 9170 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.604806 9175 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:02:59.605247 9175 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:59.608683 9175 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.611095 9171 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:59.611348 9171 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.014s
I20240501 14:02:59.612064 9176 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:59.612274 9176 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20240501 14:02:59.612288 9171 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.612658 9171 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.612741 9171 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.613042 9171 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.613512 9176 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.613930 9176 raft_consensus.cc:374] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.614056 9176 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.614295 9176 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.614477 9171 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:59.614792 9171 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:02:59.615237 9170 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:59.615782 9170 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20240501 14:02:59.616816 9170 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.617089 9060 raft_consensus.cc:482] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.617295 9060 raft_consensus.cc:504] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.617849 9177 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:59.618085 9177 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.005s
I20240501 14:02:59.618484 9060 leader_election.cc:288] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:59.619005 9177 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.619145 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:59.619364 7140 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.619419 9177 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.619606 9177 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.619876 9176 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:59.619957 9177 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.620559 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:59.620818 7667 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.620965 9176 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:59.620975 9177 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:59.621951 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:59.622187 7926 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.622229 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:59.622468 7536 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.623311 9177 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:02:59.623485 9175 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:59.623682 9175 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240501 14:02:59.623769 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:59.623979 7797 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.624523 7339 leader_election.cc:302] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.624686 9175 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.625439 9176 tablet_bootstrap.cc:654] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.626024 9060 raft_consensus.cc:2781] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.626119 9060 raft_consensus.cc:482] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.626261 9060 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.626305 9173 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:02:59.626497 9173 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.062s user 0.027s sys 0.021s
I20240501 14:02:59.627377 9173 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.621270 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:59.627758 9173 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.627857 9173 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.627910 7271 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.628173 9173 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.628932 9173 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:59.628465 9170 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.625090 9175 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.629102 9175 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.629174 9170 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.629453 9175 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.631736 9175 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:02:59.632117 9175 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:02:59.628963 9060 raft_consensus.cc:504] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.634250 9177 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.634495 9175 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.634361 9170 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.635154 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:59.635383 7797 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.636829 9170 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240501 14:02:59.637204 9170 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:02:59.638041 7797 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.633667 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:59.638338 7667 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.640996 7667 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.642730 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:59.643236 7271 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.643741 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:59.643929 7140 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.645891 7271 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.646433 7338 leader_election.cc:302] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.646641 7140 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.647408 9060 leader_election.cc:288] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:59.647874 9060 raft_consensus.cc:2781] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.648061 9175 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:02:59.648051 9060 raft_consensus.cc:686] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Running, Role: LEADER
I20240501 14:02:59.648257 9175 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240501 14:02:59.648401 9060 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.649436 9175 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.649803 9175 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.649937 9175 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.650305 9175 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.651110 9175 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:59.653482 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:59.653704 7536 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.655640 9170 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.656643 7536 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.657634 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cf39e6be52426c93c1534754be7264" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:59.657846 7926 raft_consensus.cc:3037] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.658278 9172 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:02:59.658553 9172 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.005s
I20240501 14:02:59.659046 9171 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.659427 9172 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.660775 7926 raft_consensus.cc:2454] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.660827 9172 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.661280 9172 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.661986 9172 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.663532 9172 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:02:59.667569 9177 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:02:59.667784 9177 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20240501 14:02:59.668674 9177 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.669063 9177 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.669243 9177 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.669534 9177 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.670060 9177 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:59.672294 9171 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:02:59.672451 9171 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.008s
I20240501 14:02:59.673441 9171 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.673713 9171 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.673815 9171 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.674073 9171 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.674872 9171 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:59.676878 9170 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:02:59.677039 9170 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240501 14:02:59.677266 9176 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:59.677451 9176 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.057s user 0.018s sys 0.013s
I20240501 14:02:59.677891 9170 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.678149 9170 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.678262 9170 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.678484 9170 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.679111 9170 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:59.678215 9176 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.679318 6995 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:59.679844 9176 raft_consensus.cc:374] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.679931 9176 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.680212 9176 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.680956 9176 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:59.685495 9176 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:02:59.687114 9176 tablet_bootstrap.cc:654] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:59.688650 9176 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:02:59.688781 9176 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:02:59.689424 9176 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.689622 9176 raft_consensus.cc:374] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:59.689720 9176 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:02:59.689942 9176 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.690402 9176 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:59.723769 8941 raft_consensus.cc:482] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.724043 8941 raft_consensus.cc:504] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.725219 8941 leader_election.cc:288] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:59.725348 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:59.725479 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:59.725620 7536 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.725582 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:59.725728 7926 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.725809 7797 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.725816 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:59.726066 7271 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.726356 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:59.726576 7667 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.726544 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:02:59.726743 7404 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:02:59.726924 7074 leader_election.cc:302] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.727327 8941 raft_consensus.cc:2781] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.727469 8941 raft_consensus.cc:482] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.727596 8941 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.729597 8941 raft_consensus.cc:504] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.730710 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:59.730742 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:59.730855 8941 leader_election.cc:288] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:02:59.730953 7536 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.730949 7667 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.731243 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:59.731462 7797 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.733021 7667 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.733891 7536 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.733928 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:59.734158 7926 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.734292 7797 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.734277 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:59.734503 7271 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.734689 7074 leader_election.cc:302] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.734676 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227b60efb7343bba8c4e89f81c3f6ff" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:02:59.734869 7404 raft_consensus.cc:3037] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.735061 8941 raft_consensus.cc:2781] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.736982 7926 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.737462 8941 raft_consensus.cc:686] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Running, Role: LEADER
I20240501 14:02:59.737637 7271 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.737844 8941 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.738581 7404 raft_consensus.cc:2454] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 1.
I20240501 14:02:59.742017 6993 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d05240051574799acbbe13e21b262b4 (127.22.5.193). New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:59.783957 9060 raft_consensus.cc:482] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.784170 9060 raft_consensus.cc:504] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.785292 9060 leader_election.cc:288] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:59.785336 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:02:59.785449 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:02:59.785427 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:02:59.785694 7797 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.785691 7667 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.785952 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:02:59.786017 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:02:59.786167 7271 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.786231 7140 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.786440 7536 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.786607 7338 leader_election.cc:302] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.786906 9060 raft_consensus.cc:2781] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:59.787007 9060 raft_consensus.cc:482] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:59.787112 9060 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.787211 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:02:59.787429 7926 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:02:59.789363 9060 raft_consensus.cc:504] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.790357 9060 leader_election.cc:288] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Requested vote from peers 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:02:59.790429 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:02:59.790400 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:02:59.790601 7536 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.790572 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:02:59.790623 7797 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.790740 7667 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.792661 7667 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.793463 7797 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.793443 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:02:59.793476 7536 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.793609 7271 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.794173 7339 leader_election.cc:302] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:02:59.794262 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:02:59.794481 7140 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.794438 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c87a132b49460d89c3122c3c4f2c32" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:02:59.794605 7926 raft_consensus.cc:3037] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:59.796289 7271 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.796468 7926 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.797176 7140 raft_consensus.cc:2454] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:02:59.797271 9060 raft_consensus.cc:2781] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:59.797423 9060 raft_consensus.cc:686] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Running, Role: LEADER
I20240501 14:02:59.797806 9060 consensus_queue.cc:237] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:02:59.801788 6993 catalog_manager.cc:5484] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:00.038024 9060 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.120508 8941 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.125046 9060 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.135125 8941 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.141345 9060 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.166182 9192 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.166612 9192 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.193907 9196 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.218808 9196 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:00.222584 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:00.223074 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:03:00.225881 9196 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.232723 9196 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:00.233373 9204 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:00.239493 9060 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.240510 9060 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.241179 9196 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.250368 8941 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.267247 8941 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.291126 9241 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:00.298672 9239 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 463.073 ms
time per row: 0.231537 ms
I20240501 14:03:00.937762 6993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47450:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:03:00.938645 6993 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 14:03:00.989943 7776 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.990481 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.990849 7119 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.991298 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.992679 7647 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.993093 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.993556 7646 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.993929 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.997996 7252 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.998005 7385 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.998060 7384 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.998450 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.998485 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.998595 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.999238 7777 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:00.999665 7777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.999666 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.000058 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.001473 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.001497 7253 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.001830 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.001881 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.004410 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.005550 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.005636 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.005966 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.006100 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.011803 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.012261 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.012713 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.012987 7907 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.013450 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.014504 7909 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.014925 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.015677 7515 tablet_service.cc:1467] Processing CreateTablet for tablet c5396b2071184070bcb14763e1df7b5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.016110 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.016927 7908 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.017335 7908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.017544 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.017895 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.033140 7514 tablet_service.cc:1467] Processing CreateTablet for tablet d55e6e4ce62d43838354b96c25b75086 (DEFAULT_TABLE table=rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:01.033824 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.045785 9323 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:03:01.048064 9323 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.052093 9326 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:03:01.054908 9325 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:03:01.057114 9326 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.060364 9323 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:03:01.060621 9323 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:03:01.061714 9323 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.062243 9323 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.062392 9323 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.062531 9325 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.062741 9323 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.071559 9322 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:03:01.078511 9322 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.081012 9323 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240501 14:03:01.085891 9323 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:03:01.094522 9323 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.098556 9329 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:01.101462 9324 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:03:01.103559 9324 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.106258 9326 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:03:01.106544 9326 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.006s
I20240501 14:03:01.106595 9322 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:03:01.107645 9322 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.011s
I20240501 14:03:01.107656 9326 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.108040 9326 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.105964 9323 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
I20240501 14:03:01.108237 9326 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.108397 9323 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240501 14:03:01.108587 9326 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.109432 9326 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:01.109807 9326 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:03:01.109620 9323 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.110656 9323 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.111109 9329 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.111186 9323 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.112105 9326 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.113672 9323 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.116879 9323 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:03:01.118636 9322 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.119120 9322 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.119246 9322 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.119729 9325 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:03:01.119938 9325 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.000s
I20240501 14:03:01.119800 9322 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.120990 9325 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.121307 9325 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.121435 9325 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.121724 9325 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.122049 9322 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20240501 14:03:01.122416 9322 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:03:01.122650 9325 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:01.123015 9325 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:03:01.124703 9323 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:03:01.127182 9323 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.127875 9043 raft_consensus.cc:482] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.128028 9043 raft_consensus.cc:504] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.129725 9043 leader_election.cc:288] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:01.130518 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.131441 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:03:01.133479 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.133785 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.134084 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.134366 9325 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.134626 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
W20240501 14:03:01.134768 7600 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.135725 9317 raft_consensus.cc:482] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.136194 9322 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.135921 9317 raft_consensus.cc:504] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.137812 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.139349 9329 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:03:01.139508 9329 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.003s
I20240501 14:03:01.140352 9329 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.140681 9329 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.140784 9329 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
W20240501 14:03:01.140844 7862 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.141157 7600 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.141420 7600 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.142750 9321 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
W20240501 14:03:01.144641 7602 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.144775 7602 leader_election.cc:302] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: db6b830d9bbe4451bd349c6ad6e14bab; no voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:03:01.144981 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.145071 9239 raft_consensus.cc:482] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.145211 9324 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:03:01.145247 7273 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.145362 9324 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.004s
I20240501 14:03:01.145258 9239 raft_consensus.cc:504] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.146175 9324 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.146453 9324 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.146461 9239 leader_election.cc:288] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:01.146543 9324 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.146867 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.146795 9324 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.147058 7273 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:03:01.147102 9043 raft_consensus.cc:2726] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:01.147626 9324 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:01.148039 9324 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:03:01.148208 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.148617 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.148898 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
W20240501 14:03:01.149173 7074 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.149575 7074 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.149511 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:03:01.149725 7929 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d05240051574799acbbe13e21b262b4 in term 0.
I20240501 14:03:01.149847 9324 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.149888 9329 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.148663 9322 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:03:01.150485 9322 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20240501 14:03:01.150810 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.151042 7140 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.151906 9321 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.151921 9323 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: No bootstrap required, opened a new log
W20240501 14:03:01.151995 7601 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.152065 9323 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240501 14:03:01.152173 9329 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
W20240501 14:03:01.152222 7601 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.152884 9329 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:01.152990 9323 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.153280 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "0d05240051574799acbbe13e21b262b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:03:01.153396 9323 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.153522 9323 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.154337 9322 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.156792 9322 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.156901 9322 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.157302 9322 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.158790 9329 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.159853 9317 leader_election.cc:288] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:01.160950 9321 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:03:01.160966 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.161350 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:03:01.161553 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
W20240501 14:03:01.162163 7863 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.162410 7862 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.155089 7075 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:01.162621 7862 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.162685 9321 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240501 14:03:01.162729 7862 leader_election.cc:302] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:03:01.162994 9323 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
W20240501 14:03:01.164414 7074 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.164570 7074 leader_election.cc:302] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d; no voters: 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:03:01.165427 9206 raft_consensus.cc:2726] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:01.166469 9325 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:03:01.166752 9325 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240501 14:03:01.167707 9325 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.168237 9325 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.168375 9325 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.168732 9325 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.169365 9322 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240501 14:03:01.165421 9321 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.169765 9242 raft_consensus.cc:2726] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:01.170053 9321 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.170193 9321 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.171238 9323 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240501 14:03:01.172183 9325 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:03:01.172530 9322 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d: Bootstrap starting.
I20240501 14:03:01.173043 9321 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.173563 9325 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:03:01.175151 9043 raft_consensus.cc:482] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.175276 9326 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:03:01.175458 9322 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.175316 9043 raft_consensus.cc:504] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.175536 9326 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.006s
I20240501 14:03:01.176098 9321 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:03:01.176601 9321 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:03:01.176741 9043 leader_election.cc:288] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:01.177608 9325 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.178179 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.178648 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:03:01.179319 9329 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:03:01.179325 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.179484 9329 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20240501 14:03:01.179550 7798 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
W20240501 14:03:01.180325 7600 leader_election.cc:341] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.180267 9329 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.180598 9329 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.180673 9329 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.180953 9329 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
W20240501 14:03:01.181265 7601 leader_election.cc:341] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.182483 9321 tablet_bootstrap.cc:654] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.182857 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.183133 9329 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:03:01.183128 7140 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6b830d9bbe4451bd349c6ad6e14bab in term 0.
I20240501 14:03:01.183521 9329 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:01.183712 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
W20240501 14:03:01.184334 7601 leader_election.cc:341] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.184916 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.186988 9329 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.189555 9324 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:03:01.189867 9324 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.016s
W20240501 14:03:01.191049 7600 leader_election.cc:341] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.191223 7600 leader_election.cc:302] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, db6b830d9bbe4451bd349c6ad6e14bab; no voters: 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:03:01.191576 9043 raft_consensus.cc:2726] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:01.190948 9324 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.191969 9324 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.192114 9324 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.192523 9324 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.193521 9324 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:03:01.193957 9324 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:03:01.195544 9324 tablet_bootstrap.cc:654] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.206702 9329 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210: No bootstrap required, opened a new log
I20240501 14:03:01.206904 9329 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20240501 14:03:01.207774 9329 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.208137 9329 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.208253 9329 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.208575 9321 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:03:01.208847 9321 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.002s
I20240501 14:03:01.210390 9321 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.210877 9321 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.211011 9321 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.212075 9321 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.211316 9329 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.213173 9321 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:03:01.214835 9329 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240501 14:03:01.215456 9321 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:03:01.219048 9321 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.221297 9270 raft_consensus.cc:482] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.221522 9270 raft_consensus.cc:504] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.223157 9270 leader_election.cc:288] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:01.223342 9317 raft_consensus.cc:482] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.223569 9317 raft_consensus.cc:504] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.224525 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.224527 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:03:01.224799 9317 leader_election.cc:288] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:01.224778 7404 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:03:01.224793 7535 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:03:01.225665 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.225692 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.225893 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:03:01.225962 7140 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:03:01.226122 7929 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
W20240501 14:03:01.226218 7862 leader_election.cc:341] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.226425 7733 leader_election.cc:302] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:03:01.226538 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:03:01.226766 7666 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.226737 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
W20240501 14:03:01.227475 7731 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.227903 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:03:01.228158 7535 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.228206 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.228317 9324 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13: No bootstrap required, opened a new log
I20240501 14:03:01.228463 9324 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.010s
I20240501 14:03:01.228821 9270 raft_consensus.cc:2781] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.228976 9270 raft_consensus.cc:482] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.229076 9270 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.229472 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.229717 7404 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.229578 9324 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.230103 9324 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.230134 7862 leader_election.cc:302] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters: c8f737f3f6994db297b47fb23dad1b13
I20240501 14:03:01.230250 9324 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.230509 9317 raft_consensus.cc:2781] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.230608 9317 raft_consensus.cc:482] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.230716 9317 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.230816 9324 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.231645 9324 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:01.233003 9317 raft_consensus.cc:504] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.233351 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.233600 7140 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.234704 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:03:01.236012 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:03:01.236310 7404 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.240777 7404 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:03:01.242707 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:03:01.242959 7798 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.245232 9270 raft_consensus.cc:504] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.247054 7798 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:03:01.248637 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:03:01.248929 7665 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.253744 7665 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:03:01.255744 9317 leader_election.cc:288] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Requested vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:01.261216 9326 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.261660 9326 raft_consensus.cc:374] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.261740 9326 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.262115 9326 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.262830 7271 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 0.
I20240501 14:03:01.263180 9326 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.087s user 0.003s sys 0.000s
I20240501 14:03:01.266741 9326 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc: Bootstrap starting.
I20240501 14:03:01.271945 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:03:01.272246 7140 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.273201 7862 leader_election.cc:302] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters: b3b866abc97f437e8f24eccb653b3dcc
I20240501 14:03:01.274475 9353 raft_consensus.cc:2781] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.274955 9353 raft_consensus.cc:686] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Becoming Leader. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Running, Role: LEADER
I20240501 14:03:01.275502 9353 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.276722 7140 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:03:01.280522 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:03:01.280781 7404 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.280794 9270 leader_election.cc:288] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Requested vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:01.281107 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:03:01.281384 7535 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.281380 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
W20240501 14:03:01.282246 7731 leader_election.cc:341] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:01.285193 7535 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:03:01.286480 7404 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:03:01.287052 7731 leader_election.cc:302] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters: db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:03:01.287585 9270 raft_consensus.cc:2781] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.287853 9270 raft_consensus.cc:686] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Becoming Leader. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Running, Role: LEADER
I20240501 14:03:01.288290 9270 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.298000 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:03:01.298331 7929 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.296543 6997 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198). New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:01.302688 7929 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
W20240501 14:03:01.322723 7863 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.196:37547, user_credentials={real_user=slave}} blocked reactor thread for 86855us
W20240501 14:03:01.325160 7864 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 93470.8us
I20240501 14:03:01.326040 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:03:01.326329 7140 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:03:01.329463 7208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 131038us
I20240501 14:03:01.330192 7140 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:03:01.332695 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.332986 7271 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 0.
I20240501 14:03:01.333701 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "c8f737f3f6994db297b47fb23dad1b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:03:01.333940 7271 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.337368 7271 raft_consensus.cc:2454] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f737f3f6994db297b47fb23dad1b13 in term 1.
I20240501 14:03:01.338474 9326 tablet_bootstrap.cc:654] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:01.340320 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5396b2071184070bcb14763e1df7b5f" candidate_uuid: "cc7e7386a9e540eabc4648e147073f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:03:01.340569 7535 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.343318 7535 raft_consensus.cc:2454] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7e7386a9e540eabc4648e147073f1d in term 1.
I20240501 14:03:01.346359 9325 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab: No bootstrap required, opened a new log
I20240501 14:03:01.346402 9326 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc: No bootstrap required, opened a new log
I20240501 14:03:01.346527 9325 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.173s user 0.001s sys 0.061s
I20240501 14:03:01.346618 9326 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.001s
I20240501 14:03:01.347391 9321 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: No bootstrap required, opened a new log
I20240501 14:03:01.347285 9325 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.347563 9325 raft_consensus.cc:374] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.347571 9321 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.044s
I20240501 14:03:01.347647 9325 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.347785 9326 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.347877 9325 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.348269 9321 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.348459 9325 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:01.348526 9321 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.348625 9321 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.348861 9321 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.349673 9321 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:01.351136 9326 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.352270 9326 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.352643 9326 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.353822 9326 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:03:01.343253 6997 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7e7386a9e540eabc4648e147073f1d (127.22.5.199). New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:01.358178 9322 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d: No bootstrap required, opened a new log
I20240501 14:03:01.358366 9322 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d: Time spent bootstrapping tablet: real 0.186s user 0.064s sys 0.061s
I20240501 14:03:01.359195 9322 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.359493 9322 raft_consensus.cc:374] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:01.359611 9322 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e7386a9e540eabc4648e147073f1d, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.359850 9322 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.360386 9322 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:01.554525 9192 raft_consensus.cc:482] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.554721 9192 raft_consensus.cc:504] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.555867 9192 leader_election.cc:288] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:01.556087 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" is_pre_election: true
I20240501 14:03:01.556234 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:03:01.556203 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:03:01.556370 7665 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.556450 7929 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.556471 7536 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.556573 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.556855 7798 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.557041 7339 leader_election.cc:302] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 704fb2d37343406790224dc76b40f210, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:03:01.557103 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.557370 7271 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.557433 9192 raft_consensus.cc:2781] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:01.557530 9192 raft_consensus.cc:482] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:01.557509 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.557618 9192 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.557775 7140 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 0.
I20240501 14:03:01.559677 9192 raft_consensus.cc:504] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.560925 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:03:01.561044 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6b830d9bbe4451bd349c6ad6e14bab"
I20240501 14:03:01.561174 7929 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.561292 7665 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.562407 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:03:01.562418 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:03:01.562613 7798 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.562633 7536 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.564155 7665 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.564527 7929 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.565148 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:03:01.565385 7798 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.565405 7140 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.566040 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e6e4ce62d43838354b96c25b75086" candidate_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
I20240501 14:03:01.566248 7271 raft_consensus.cc:3037] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.566263 7338 leader_election.cc:302] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: c8f737f3f6994db297b47fb23dad1b13, cc7e7386a9e540eabc4648e147073f1d, ceb055f6a1a540dcb61e1e11a157cdb8, db6b830d9bbe4451bd349c6ad6e14bab; no voters:
I20240501 14:03:01.566520 7536 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.567166 9192 leader_election.cc:288] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [CANDIDATE]: Term 1 election: Requested vote from peers c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:01.567633 9192 raft_consensus.cc:2781] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:01.567778 9192 raft_consensus.cc:686] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Running, Role: LEADER
I20240501 14:03:01.568153 7140 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.568312 9192 consensus_queue.cc:237] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:01.569391 7271 raft_consensus.cc:2454] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb055f6a1a540dcb61e1e11a157cdb8 in term 1.
I20240501 14:03:01.572347 6997 catalog_manager.cc:5484] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195). New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:01.670300 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.726014 9362 raft_consensus.cc:482] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.726224 9362 raft_consensus.cc:504] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:01.727419 9362 leader_election.cc:288] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451), 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:01.727684 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d05240051574799acbbe13e21b262b4" is_pre_election: true
I20240501 14:03:01.728080 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc" is_pre_election: true
I20240501 14:03:01.728243 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f737f3f6994db297b47fb23dad1b13" is_pre_election: true
I20240501 14:03:01.728395 7271 raft_consensus.cc:2379] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db6b830d9bbe4451bd349c6ad6e14bab in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:03:01.728756 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" is_pre_election: true
I20240501 14:03:01.729048 7404 raft_consensus.cc:2379] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db6b830d9bbe4451bd349c6ad6e14bab in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:03:01.729506 7600 leader_election.cc:302] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: db6b830d9bbe4451bd349c6ad6e14bab; no voters: 0d05240051574799acbbe13e21b262b4, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:03:01.729614 7536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704fb2d37343406790224dc76b40f210" is_pre_election: true
I20240501 14:03:01.729840 9362 raft_consensus.cc:3037] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.729884 7536 raft_consensus.cc:2379] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db6b830d9bbe4451bd349c6ad6e14bab in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:03:01.729785 7929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2a7bfaac904352a9c9c12c09b563f5" candidate_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e7386a9e540eabc4648e147073f1d" is_pre_election: true
I20240501 14:03:01.730073 7929 raft_consensus.cc:2379] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db6b830d9bbe4451bd349c6ad6e14bab in current term 1: Already voted for candidate c8f737f3f6994db297b47fb23dad1b13 in this term.
I20240501 14:03:01.732492 9362 raft_consensus.cc:2726] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:01.734050 9317 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.742453 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.767537 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.768734 7271 raft_consensus.cc:3037] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:01.774545 9317 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.777782 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.779273 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.784570 9317 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.791606 9330 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.820228 9206 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.861968 9351 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.864001 9317 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.966291 9192 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:01.974578 9360 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.066695 9360 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.073307 9391 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.078100 9360 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:02.089455 9391 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:02.121368 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:02.121717 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:02.129778 9363 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 437.787 ms
time per row: 0.218893 ms
I20240501 14:03:02.729095 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.200:43849
--local_ip_for_outbound_sockets=127.22.5.200
--tserver_master_addrs=127.22.5.254:41683
--webserver_port=42129
--webserver_interface=127.22.5.200
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:03:03.131793 9461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:03.132243 9461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:03.132421 9461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:03.141345 9461 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:03.141491 9461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:03.141610 9461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.200
I20240501 14:03:03.151484 9461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.200:43849
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.5.200
--webserver_port=42129
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.200
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:03.152861 9461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:03.154659 9461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:03.168550 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:03.169153 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:03.169690 9468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:03.170372 9461 server_base.cc:1034] running on GCE node
I20240501 14:03:03.171106 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:03.172448 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:03.173651 9461 hybrid_clock.cc:648] HybridClock initialized: now 1714572183173586 us; error 65 us; skew 500 ppm
I20240501 14:03:03.173982 9461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:03.180505 9461 webserver.cc:460] Webserver started at http://127.22.5.200:42129/ using document root <none> and password file <none>
I20240501 14:03:03.181437 9461 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:03.181617 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:03.187880 9461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 14:03:03.190984 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.192524 9461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:03.192719 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "cc3a47f051a546a5a7788d088f061dd5"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:03:03.193867 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:03.227875 9461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:03.228717 9461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:03.228967 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:03.229946 9461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:03.231485 9461 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:03.231563 9461 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.231679 9461 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:03.231724 9461 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.270068 9461 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.200:43849
I20240501 14:03:03.270126 9588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.200:43849 every 8 connection(s)
I20240501 14:03:03.271703 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:03:03.278759 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 9461
I20240501 14:03:03.279393 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.201:33735
--local_ip_for_outbound_sockets=127.22.5.201
--tserver_master_addrs=127.22.5.254:41683
--webserver_port=39661
--webserver_interface=127.22.5.201
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:03.287351 9589 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:03:03.287729 9589 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:03.288787 9589 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:03:03.290843 7000 ts_manager.cc:194] Re-registered known tserver with Master: cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849)
I20240501 14:03:03.291644 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.200:39881
W20240501 14:03:03.693056 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:03.693413 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:03.693545 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:03.702142 9593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:03.702278 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:03.702410 9593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.201
I20240501 14:03:03.712839 9593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.201:33735
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.22.5.201
--webserver_port=39661
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.201
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:03.714558 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:03.717206 9593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:03.730749 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:03.731385 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:03.731671 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:03.732470 9593 server_base.cc:1034] running on GCE node
I20240501 14:03:03.733152 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:03.734194 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:03.735412 9593 hybrid_clock.cc:648] HybridClock initialized: now 1714572183735352 us; error 65 us; skew 500 ppm
I20240501 14:03:03.735750 9593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:03.738363 9593 webserver.cc:460] Webserver started at http://127.22.5.201:39661/ using document root <none> and password file <none>
I20240501 14:03:03.739384 9593 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:03.739547 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:03.745905 9593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:03:03.749099 9610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.750701 9593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:03:03.750860 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "41ab226fa6b7403b8b44642e9470517d"
format_stamp: "Formatted at 2024-05-01 14:02:45 on dist-test-slave-5jzj"
I20240501 14:03:03.751754 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:03.779950 9593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:03.780864 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:03.781212 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:03.782408 9593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:03.784060 9593 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:03.784132 9593 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.784262 9593 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:03.784310 9593 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:03.831251 9593 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.201:33735
I20240501 14:03:03.831385 9722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.201:33735 every 8 connection(s)
I20240501 14:03:03.832890 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 14:03:03.839244 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 9593
I20240501 14:03:03.840065 22551 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
/tmp/dist-test-taskA1ASV2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.5.202:37399
--local_ip_for_outbound_sockets=127.22.5.202
--tserver_master_addrs=127.22.5.254:41683
--webserver_port=33341
--webserver_interface=127.22.5.202
--builtin_ntp_servers=127.22.5.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:03.847728 9723 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:03:03.848068 9723 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:03.848939 9723 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:03:03.850538 7000 ts_manager.cc:194] Re-registered known tserver with Master: 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735)
I20240501 14:03:03.851318 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.201:40237
W20240501 14:03:04.255340 9727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:04.255730 9727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:04.255898 9727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:04.264197 9727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:04.264331 9727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:04.264506 9727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.5.202
I20240501 14:03:04.277166 9727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.5.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.5.202:37399
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.22.5.202
--webserver_port=33341
--tserver_master_addrs=127.22.5.254:41683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.5.202
--log_dir=/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:04.278527 9727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:04.280316 9727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:03:04.294757 9589 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
W20240501 14:03:04.294955 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:04.295485 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:04.296245 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:04.297084 9727 server_base.cc:1034] running on GCE node
I20240501 14:03:04.297775 9727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:04.298861 9727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:04.300077 9727 hybrid_clock.cc:648] HybridClock initialized: now 1714572184300018 us; error 56 us; skew 500 ppm
I20240501 14:03:04.300434 9727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:04.302856 9727 webserver.cc:460] Webserver started at http://127.22.5.202:33341/ using document root <none> and password file <none>
I20240501 14:03:04.303654 9727 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:04.303781 9727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:04.311235 9727 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:03:04.314782 9746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:04.316273 9727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:03:04.316447 9727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1c61c125cd144590ae1e54a2f87d9187"
format_stamp: "Formatted at 2024-05-01 14:02:46 on dist-test-slave-5jzj"
I20240501 14:03:04.317349 9727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:04.351507 9727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:04.352537 9727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:04.352839 9727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:04.353964 9727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:04.355482 9727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:04.355552 9727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:04.355654 9727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:04.355726 9727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:04.403623 9727 rpc_server.cc:307] RPC server started. Bound to: 127.22.5.202:37399
I20240501 14:03:04.403627 9858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.5.202:37399 every 8 connection(s)
I20240501 14:03:04.405447 9727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240501 14:03:04.407181 22551 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA1ASV2/build/asan/bin/kudu as pid 9727
I20240501 14:03:04.452694 9859 heartbeater.cc:347] Connected to a master server at 127.22.5.254:41683
I20240501 14:03:04.453080 9859 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:04.454169 9859 heartbeater.cc:511] Master 127.22.5.254:41683 requested a full tablet report, sending...
I20240501 14:03:04.455929 7000 ts_manager.cc:194] Re-registered known tserver with Master: 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399)
I20240501 14:03:04.466272 7000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.5.202:58873
I20240501 14:03:04.854000 9723 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
I20240501 14:03:05.066220 7120 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.069414 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.083722 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:05.105031 7251 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.154574 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.176784 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:05.217547 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.231150 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.249660 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:05.284801 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:05.470018 9859 heartbeater.cc:503] Master 127.22.5.254:41683 was elected leader, sending a full tablet report...
I20240501 14:03:05.527686 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.534498 7120 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.537907 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:05.570431 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.589537 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:05.600381 7251 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.607565 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.612792 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:05.613282 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.658703 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:05.927943 7120 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:05.984372 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.001142 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.013689 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:06.015003 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:06.022017 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.045722 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:06.058220 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:06.071516 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:06.073275 7251 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:06.216148 7404 consensus_queue.cc:237] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:06.219494 7929 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.219502 7139 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.220594 9392 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.220968 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.221706 7536 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.222517 9595 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.225513 7666 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.225571 7798 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.226771 9595 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.227613 7272 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.227612 10016 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.228525 10016 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.235057 10016 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.235975 7929 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.244024 7272 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.244740 7666 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.246192 6999 catalog_manager.cc:5484] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.251607 7536 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.256104 7798 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.269474 7139 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.271347 7272 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:06.275085 7929 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.275782 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.275954 7536 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.276054 7798 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.276700 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.277271 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.299489 7139 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.300629 10037 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.305711 7666 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.308185 7404 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.309080 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.310237 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.321156 9349 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.329602 7139 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.330991 7798 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.341373 7536 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.341390 7929 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.347499 7402 consensus_queue.cc:237] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:06.350147 6993 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.355122 7665 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.355790 10018 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.358472 7536 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.358817 7929 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.359000 10018 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.360045 10018 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.364689 7798 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.365103 10018 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.367331 10016 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.372865 7535 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.373751 7272 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.375074 9392 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.377287 7272 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:06.380761 7271 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.387291 7536 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.387987 10039 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.388475 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.390208 7140 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.390602 9349 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.392462 7404 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.393123 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.394352 6992 catalog_manager.cc:5484] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.394685 7141 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.398445 7667 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.399222 7927 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.399971 10038 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.400991 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 11c87a132b49460d89c3122c3c4f2c32 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.405854 9349 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.411335 7141 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.415064 7927 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.417969 7535 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.421465 7929 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.422351 6992 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:06.430344 7665 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:03:06.431421 7341 consensus_peers.cc:485] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11c87a132b49460d89c3122c3c4f2c32. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.431955 7798 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.452848 7798 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.454353 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 253a0abd26814fb29f9adede664c5988 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.454979 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.456791 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.458765 7665 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.459838 7404 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.464809 7404 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.466472 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.469115 7798 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:06.474303 7206 consensus_peers.cc:485] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253a0abd26814fb29f9adede664c5988. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.486923 7665 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.510363 7665 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } }
I20240501 14:03:06.513890 7929 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.514106 7139 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.515280 10052 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.516198 7272 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.516253 7404 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.517087 10052 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.517510 10054 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.518613 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.522742 7798 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.522954 7535 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.523679 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.524153 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.535066 10054 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.536260 7929 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.537492 7139 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.542214 6995 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:06.544523 7798 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.546545 7404 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.548746 7272 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.552747 7535 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } }
I20240501 14:03:06.565791 7665 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:06.571190 7141 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.571830 7929 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.572539 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.572974 10054 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.574622 7535 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.575186 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.575718 7798 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.576411 7404 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.576457 7272 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.577315 10052 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.577821 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.578377 9362 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.580184 10052 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.582170 7404 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.585613 7535 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.585747 7798 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.588673 7272 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.590054 7000 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.591823 7927 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:06.603933 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 210980fcec0c4358943ddd2cc7e935a6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.604691 7798 consensus_queue.cc:237] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:06.608203 7927 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.608597 7665 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.608824 7404 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.609381 7272 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.609402 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.609714 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.609797 7141 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.610330 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.611413 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.612072 7535 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.612653 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:03:06.619266 7603 consensus_peers.cc:485] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 210980fcec0c4358943ddd2cc7e935a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.620256 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.643072 9729 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.644680 7927 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.645790 7404 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.646284 7141 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.647670 7272 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.651883 6991 catalog_manager.cc:5484] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.658694 7535 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.659006 7665 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:06.671018 7798 consensus_queue.cc:237] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:06.674367 7141 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.676098 7536 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.676347 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.677063 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.680070 7665 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.680748 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.681596 7404 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.682262 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.684052 10074 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.684762 7665 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.687997 7927 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.689149 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f920ec86006647cc8e4efbf88dabd6cc with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.691352 10035 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:03:06.691851 7731 consensus_peers.cc:485] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f920ec86006647cc8e4efbf88dabd6cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.692868 7139 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.693243 7000 catalog_manager.cc:5484] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.693110 7927 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.693406 7536 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.695125 7272 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.696102 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.696827 7405 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.705271 7271 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:06.716732 7271 consensus_queue.cc:237] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:06.720332 7139 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.720860 10038 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.721850 7796 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.722537 10038 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.723398 7665 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.723598 7536 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.724375 7927 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.724927 9349 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.725622 9349 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.725749 7404 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.729189 10092 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.730592 10038 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.745505 10037 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.746245 7536 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.746315 7139 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.751199 6991 catalog_manager.cc:5484] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.752794 7796 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.754609 7665 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.756855 7404 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.762632 7927 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.772436 7271 consensus_queue.cc:237] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:06.776050 7665 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.776443 7536 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.776942 10092 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.777969 10033 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.780521 7139 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.781203 7404 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.781816 10037 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.782413 10037 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:03:06.783165 7206 consensus_peers.cc:485] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c89a8bb1e4c499f9619676cc6af97d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.786499 10038 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.789855 7927 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.790760 7141 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.794061 10039 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.795276 7536 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.795912 7927 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.800669 7404 consensus_queue.cc:237] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:06.802381 6998 catalog_manager.cc:5484] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.803146 7796 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.804464 10033 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.806545 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4c89a8bb1e4c499f9619676cc6af97d2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.807569 7666 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.810565 7796 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.815737 7141 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.816344 10018 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.823012 7927 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.823189 7666 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.823381 7536 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.824177 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.824502 10018 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.824607 7271 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.825235 10047 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.825726 7796 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.825668 10017 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.827260 10017 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.829560 10018 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.830116 7665 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.830214 7404 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.837150 7140 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.842422 7927 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.843850 6998 catalog_manager.cc:5484] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:06.848824 7536 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.851012 7796 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.854352 7404 consensus_queue.cc:237] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:06.859695 7140 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.860368 10017 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.861624 7927 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.861644 7536 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.863606 10016 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.864248 10016 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.866426 7796 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.867444 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.868137 7665 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.869029 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.870661 7271 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.871207 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.872433 7271 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.872583 9392 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:06.874497 7338 consensus_peers.cc:485] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55e6e4ce62d43838354b96c25b75086. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:06.877372 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d55e6e4ce62d43838354b96c25b75086 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:06.877475 7536 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.878613 7796 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.878777 7927 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.881860 7272 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.884265 6995 catalog_manager.cc:5484] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.884759 7666 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.885900 7140 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:06.905426 10120 ts_tablet_manager.cc:902] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:06.911837 7796 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:06.915529 7404 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.916296 7140 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.916441 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.917610 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.922360 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.923099 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.925755 7927 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.926366 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.939273 7666 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.939968 7536 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:06.940524 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.940847 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:06.976815 10074 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.978219 7665 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.986878 7000 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:06.990584 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:07.000921 7272 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:07.014809 7796 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.017048 7536 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:07.020700 7666 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.021752 7140 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.022194 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.024559 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.026090 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.027429 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.028775 7927 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
W20240501 14:03:07.032922 7734 consensus_peers.cc:485] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64b9b1b64def483aaed2a03f7d356915. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:07.034788 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.034271 7141 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.038552 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:03:07.042454 10084 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.045806 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.049962 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 64b9b1b64def483aaed2a03f7d356915 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:07.050494 7666 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.052338 7139 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.055958 6995 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:07.084605 7927 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.097632 10120 tablet_copy_client.cc:308] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:07.101917 10135 ts_tablet_manager.cc:902] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:07.109201 7405 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.110499 10084 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.112318 7405 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:07.118525 7405 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.122081 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 11c87a132b49460d89c3122c3c4f2c32 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:46003)
I20240501 14:03:07.122429 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 11c87a132b49460d89c3122c3c4f2c32 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:46003: session id = 41ab226fa6b7403b8b44642e9470517d-11c87a132b49460d89c3122c3c4f2c32
I20240501 14:03:07.127295 7425 tablet_copy_source_session.cc:205] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.137885 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c87a132b49460d89c3122c3c4f2c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.150570 7535 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.152175 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.157302 7535 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:07.164997 10133 ts_tablet_manager.cc:902] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:07.170609 10135 tablet_copy_client.cc:308] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:07.172720 7536 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:07.173465 10120 tablet_copy_client.cc:783] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.173867 10120 tablet_copy_client.cc:647] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.176808 10120 tablet_copy_client.cc:523] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.182374 10120 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:07.183542 7927 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.184171 7140 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.184346 9728 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.184449 7405 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.185106 9728 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.185992 7796 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.185061 7666 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.187371 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.187546 7271 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:07.189934 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.190568 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.191179 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.204550 9728 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.213701 7666 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.216607 7927 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.225580 7000 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.226500 7405 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.231925 7271 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.232928 10120 log.cc:824] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:07.238039 7536 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.244884 10133 tablet_copy_client.cc:308] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:42421
I20240501 14:03:07.245965 10162 ts_tablet_manager.cc:902] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:07.256451 7405 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.257419 7927 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.258150 10153 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.264271 7665 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.272615 7687 tablet_copy_service.cc:140] P db6b830d9bbe4451bd349c6ad6e14bab: Received BeginTabletCopySession request for tablet 210980fcec0c4358943ddd2cc7e935a6 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:39769)
I20240501 14:03:07.272861 7687 tablet_copy_service.cc:161] P db6b830d9bbe4451bd349c6ad6e14bab: Beginning new tablet copy session on tablet 210980fcec0c4358943ddd2cc7e935a6 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:39769: session id = 41ab226fa6b7403b8b44642e9470517d-210980fcec0c4358943ddd2cc7e935a6
I20240501 14:03:07.276639 7687 tablet_copy_source_session.cc:205] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.278764 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.280623 7139 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.281673 10151 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.283000 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 253a0abd26814fb29f9adede664c5988 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:60957)
I20240501 14:03:07.283249 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 253a0abd26814fb29f9adede664c5988 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:60957: session id = cc3a47f051a546a5a7788d088f061dd5-253a0abd26814fb29f9adede664c5988
I20240501 14:03:07.283174 7139 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.287098 7293 tablet_copy_source_session.cc:205] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.286226 10133 tablet_copy_client.cc:783] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.291149 10133 tablet_copy_client.cc:647] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.296754 10133 tablet_copy_client.cc:523] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.298084 10135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.300335 7271 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.300734 10133 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:07.301667 10151 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.302830 7796 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:07.304471 10151 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:07.312144 10151 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:07.312425 10120 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.313174 10120 tablet_bootstrap.cc:492] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:07.313778 10120 ts_tablet_manager.cc:1367] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.132s user 0.059s sys 0.013s
I20240501 14:03:07.314254 7796 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:07.317732 10151 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240501 14:03:07.319424 10140 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.322924 7405 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.324647 7666 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.325932 10135 tablet_copy_client.cc:783] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.322283 10162 tablet_copy_client.cc:308] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:07.326462 10135 tablet_copy_client.cc:647] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.330071 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f56b1674c3ec4bdc9c2bd82a231d0112 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:07.330277 10135 tablet_copy_client.cc:523] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.332345 6995 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.336786 10135 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:07.337190 7927 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.337499 10120 raft_consensus.cc:348] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.338101 10120 raft_consensus.cc:729] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:07.338750 10120 consensus_queue.cc:260] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.343973 7139 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.352260 7271 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:07.364449 7798 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:07.380972 7469 consensus_peers.cc:485] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f56b1674c3ec4bdc9c2bd82a231d0112. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:07.395805 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet f920ec86006647cc8e4efbf88dabd6cc from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:42055)
I20240501 14:03:07.396095 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet f920ec86006647cc8e4efbf88dabd6cc from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:42055: session id = 1c61c125cd144590ae1e54a2f87d9187-f920ec86006647cc8e4efbf88dabd6cc
I20240501 14:03:07.399833 10120 ts_tablet_manager.cc:1398] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.086s user 0.013s sys 0.018s
I20240501 14:03:07.400847 7818 tablet_copy_source_session.cc:205] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.402108 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-11c87a132b49460d89c3122c3c4f2c32 received from {username='slave'} at 127.22.5.201:46003
I20240501 14:03:07.402277 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-11c87a132b49460d89c3122c3c4f2c32 on tablet 11c87a132b49460d89c3122c3c4f2c32 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:07.407007 10162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920ec86006647cc8e4efbf88dabd6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.408363 10179 ts_tablet_manager.cc:902] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:07.413894 10178 ts_tablet_manager.cc:902] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:07.428573 10135 log.cc:824] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:07.429891 10179 tablet_copy_client.cc:308] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:07.442466 10133 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.442991 10133 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:07.443267 10133 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.143s user 0.065s sys 0.006s
I20240501 14:03:07.444211 10133 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.444540 10133 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:07.444763 10133 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.448201 10133 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:03:07.449019 7687 tablet_copy_service.cc:342] P db6b830d9bbe4451bd349c6ad6e14bab: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-210980fcec0c4358943ddd2cc7e935a6 received from {username='slave'} at 127.22.5.201:39769
I20240501 14:03:07.449226 7687 tablet_copy_service.cc:433] P db6b830d9bbe4451bd349c6ad6e14bab: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-210980fcec0c4358943ddd2cc7e935a6 on tablet 210980fcec0c4358943ddd2cc7e935a6 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:07.450300 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet d55e6e4ce62d43838354b96c25b75086 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:33293)
I20240501 14:03:07.450479 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet d55e6e4ce62d43838354b96c25b75086 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:33293: session id = cc3a47f051a546a5a7788d088f061dd5-d55e6e4ce62d43838354b96c25b75086
I20240501 14:03:07.452471 10178 tablet_copy_client.cc:308] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:07.453300 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 4c89a8bb1e4c499f9619676cc6af97d2 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:60957)
I20240501 14:03:07.453470 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 4c89a8bb1e4c499f9619676cc6af97d2 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:60957: session id = cc3a47f051a546a5a7788d088f061dd5-4c89a8bb1e4c499f9619676cc6af97d2
I20240501 14:03:07.453730 7425 tablet_copy_source_session.cc:205] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.456382 7293 tablet_copy_source_session.cc:205] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.458020 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e6e4ce62d43838354b96c25b75086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.460492 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c89a8bb1e4c499f9619676cc6af97d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.470636 10162 tablet_copy_client.cc:783] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.470994 10162 tablet_copy_client.cc:647] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.471189 10179 tablet_copy_client.cc:783] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.471633 10179 tablet_copy_client.cc:647] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.476626 10178 tablet_copy_client.cc:783] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.476946 10178 tablet_copy_client.cc:647] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.478078 9678 raft_consensus.cc:1203] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:07.486157 10178 tablet_copy_client.cc:523] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.486250 10162 tablet_copy_client.cc:523] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.488399 10179 tablet_copy_client.cc:523] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.489894 10178 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:07.494228 10179 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:07.501291 10162 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:07.514159 10133 ts_tablet_manager.cc:902] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:07.533036 10162 log.cc:824] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:07.537387 10133 tablet_copy_client.cc:308] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:07.545020 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet 64b9b1b64def483aaed2a03f7d356915 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:49949)
I20240501 14:03:07.545238 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet 64b9b1b64def483aaed2a03f7d356915 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:49949: session id = 41ab226fa6b7403b8b44642e9470517d-64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:07.548198 7818 tablet_copy_source_session.cc:205] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.549626 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.556401 10133 tablet_copy_client.cc:783] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.556692 10133 tablet_copy_client.cc:647] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.560545 10133 tablet_copy_client.cc:523] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.564267 10133 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:07.584933 10178 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.585608 10178 tablet_bootstrap.cc:492] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:07.586269 10178 ts_tablet_manager.cc:1367] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.096s user 0.056s sys 0.002s
I20240501 14:03:07.594031 10135 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.594331 10178 raft_consensus.cc:348] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.599893 10178 raft_consensus.cc:729] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:07.600693 10178 consensus_queue.cc:260] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.601680 10135 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:07.602010 10135 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.266s user 0.064s sys 0.008s
I20240501 14:03:07.603343 10135 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.604478 10135 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:07.605352 10135 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.614936 10178 ts_tablet_manager.cc:1398] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.028s user 0.009s sys 0.006s
I20240501 14:03:07.615808 10135 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20240501 14:03:07.618029 7292 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-253a0abd26814fb29f9adede664c5988 received from {username='slave'} at 127.22.5.200:60957
I20240501 14:03:07.618227 7292 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-253a0abd26814fb29f9adede664c5988 on tablet 253a0abd26814fb29f9adede664c5988 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:07.618791 7293 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-4c89a8bb1e4c499f9619676cc6af97d2 received from {username='slave'} at 127.22.5.200:60957
I20240501 14:03:07.618928 7293 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-4c89a8bb1e4c499f9619676cc6af97d2 on tablet 4c89a8bb1e4c499f9619676cc6af97d2 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:07.624994 10162 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.626084 10162 tablet_bootstrap.cc:492] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:07.626607 10162 ts_tablet_manager.cc:1367] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.126s user 0.065s sys 0.000s
I20240501 14:03:07.628510 10179 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.629230 10179 tablet_bootstrap.cc:492] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:07.629575 10179 ts_tablet_manager.cc:1367] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.135s user 0.038s sys 0.019s
I20240501 14:03:07.630545 10179 raft_consensus.cc:348] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.631050 10179 raft_consensus.cc:729] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:07.631405 10179 consensus_queue.cc:260] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.632321 10179 ts_tablet_manager.cc:1398] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:07.633263 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-d55e6e4ce62d43838354b96c25b75086 received from {username='slave'} at 127.22.5.200:33293
I20240501 14:03:07.633401 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-d55e6e4ce62d43838354b96c25b75086 on tablet d55e6e4ce62d43838354b96c25b75086 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:07.637245 10162 raft_consensus.cc:348] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:07.637931 10162 raft_consensus.cc:729] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:07.639065 10162 consensus_queue.cc:260] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:07.645987 10162 ts_tablet_manager.cc:1398] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.019s user 0.012s sys 0.001s
I20240501 14:03:07.646965 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-f920ec86006647cc8e4efbf88dabd6cc received from {username='slave'} at 127.22.5.202:42055
I20240501 14:03:07.647143 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-f920ec86006647cc8e4efbf88dabd6cc on tablet f920ec86006647cc8e4efbf88dabd6cc with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:07.661971 10133 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.662578 10133 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:07.662932 10133 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.099s user 0.051s sys 0.000s
I20240501 14:03:07.664067 10133 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.664505 10133 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:07.664800 10133 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:07.665786 10133 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:03:07.666648 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-64b9b1b64def483aaed2a03f7d356915 received from {username='slave'} at 127.22.5.201:49949
I20240501 14:03:07.666788 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-64b9b1b64def483aaed2a03f7d356915 on tablet 64b9b1b64def483aaed2a03f7d356915 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:07.726342 10179 ts_tablet_manager.cc:902] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547)
I20240501 14:03:07.727484 10179 tablet_copy_client.cc:308] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37547
I20240501 14:03:07.733109 7556 tablet_copy_service.cc:140] P 704fb2d37343406790224dc76b40f210: Received BeginTabletCopySession request for tablet f56b1674c3ec4bdc9c2bd82a231d0112 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:53751)
I20240501 14:03:07.733373 7556 tablet_copy_service.cc:161] P 704fb2d37343406790224dc76b40f210: Beginning new tablet copy session on tablet f56b1674c3ec4bdc9c2bd82a231d0112 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:53751: session id = cc3a47f051a546a5a7788d088f061dd5-f56b1674c3ec4bdc9c2bd82a231d0112
I20240501 14:03:07.736523 7556 tablet_copy_source_session.cc:205] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:07.738252 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:07.742986 10179 tablet_copy_client.cc:783] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:07.743232 10179 tablet_copy_client.cc:647] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:07.745388 10179 tablet_copy_client.cc:523] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:07.747891 10179 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:07.770386 9814 raft_consensus.cc:1203] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:07.780988 10179 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:07.781420 10179 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:07.781663 10179 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.011s
I20240501 14:03:07.782296 10179 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.782529 10179 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:07.782748 10179 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:07.783422 10179 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:07.785879 7556 tablet_copy_service.cc:342] P 704fb2d37343406790224dc76b40f210: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-f56b1674c3ec4bdc9c2bd82a231d0112 received from {username='slave'} at 127.22.5.200:53751
I20240501 14:03:07.786070 7556 tablet_copy_service.cc:433] P 704fb2d37343406790224dc76b40f210: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-f56b1674c3ec4bdc9c2bd82a231d0112 on tablet f56b1674c3ec4bdc9c2bd82a231d0112 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:07.820734 9544 raft_consensus.cc:1203] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:07.840198 9678 raft_consensus.cc:1203] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:07.893707 9544 raft_consensus.cc:1203] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:08.012920 10071 raft_consensus.cc:1050] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:08.014397 10071 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:08.017645 9678 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.017927 7140 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.018329 7271 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.018471 7405 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.018592 7927 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.019477 10283 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.019856 10072 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.020165 10284 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.020547 10285 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.021015 10282 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.023281 7536 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.024868 10282 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.035063 10072 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.036798 9678 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.038056 7796 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.038865 10286 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.040308 7140 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.042110 6993 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.043628 7536 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.047690 7927 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.048070 7271 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.053917 7405 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.065342 7796 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.077611 7666 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:08.080166 9678 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.080555 7271 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.081036 7927 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.081071 10284 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.081403 10071 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.081856 10071 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.082087 7536 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.082801 10071 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.082911 7140 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.084523 10071 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.084736 7405 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.085657 10071 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.086315 10283 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.086833 7271 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.089200 7402 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.090682 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 210980fcec0c4358943ddd2cc7e935a6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.091745 9678 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.092237 7927 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.093065 7140 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.097168 7535 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.097515 6999 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.102934 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet 210980fcec0c4358943ddd2cc7e935a6 with delete_type TABLET_DATA_TOMBSTONED (TS c8f737f3f6994db297b47fb23dad1b13 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33072
I20240501 14:03:08.103915 9544 raft_consensus.cc:1203] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:08.103993 10296 tablet_replica.cc:330] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: stopping tablet replica
I20240501 14:03:08.104506 10296 raft_consensus.cc:2229] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:08.104869 10296 raft_consensus.cc:2258] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:08.112843 10296 ts_tablet_manager.cc:1875] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:08.117486 10296 ts_tablet_manager.cc:1888] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:08.117645 10296 log.cc:1196] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/210980fcec0c4358943ddd2cc7e935a6
I20240501 14:03:08.118602 6984 catalog_manager.cc:4830] TS c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): tablet 210980fcec0c4358943ddd2cc7e935a6 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:08.136795 9678 raft_consensus.cc:1203] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:08.254336 9729 raft_consensus.cc:1050] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:08.255437 9729 consensus_queue.cc:237] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:08.259344 9814 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.259737 7535 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.259999 7666 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.260581 7402 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.260655 10082 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.261051 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.262322 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.262822 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.267573 7927 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.268226 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.268844 7140 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.269619 10322 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.273649 7271 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.273412 10082 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.275211 9729 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.274713 9814 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.278759 7139 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.282955 7535 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.286909 6992 catalog_manager.cc:5484] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.292734 9544 raft_consensus.cc:1203] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:08.297894 7927 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.306947 7666 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.309782 7402 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.311369 10106 raft_consensus.cc:1050] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:08.312911 10106 consensus_queue.cc:237] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.319890 7405 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.320580 7926 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.321260 10107 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.321449 7140 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.323352 10107 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.325047 7271 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.330130 10033 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.365922 9544 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.366029 7535 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.366641 7796 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.367542 10107 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.368176 10107 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.368959 10033 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.386556 10033 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.387537 9544 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.392501 7666 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.393309 6995 catalog_manager.cc:5484] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.393072 7405 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.396282 7140 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.396899 7927 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.391098 7796 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.402482 10107 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.408715 7536 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.412112 10017 raft_consensus.cc:1050] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:08.413640 10017 consensus_queue.cc:237] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.416756 10016 raft_consensus.cc:1050] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:08.425277 10016 consensus_queue.cc:237] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:08.427718 7271 consensus_queue.cc:237] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.430306 7405 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.431383 9544 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.431506 10334 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.431785 7927 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.431965 10334 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.432087 7796 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.432905 10106 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.434406 7535 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.435174 7139 raft_consensus.cc:1261] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.435422 10334 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.435914 10033 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.436178 10106 consensus_queue.cc:1035] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.440400 7927 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.441287 7536 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.441890 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.442430 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.446296 7667 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.446290 7798 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.446843 9544 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.446944 7139 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.447611 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.448122 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.448604 9392 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.451416 7271 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.457525 7666 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.462745 10107 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.470855 7534 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.473524 6995 catalog_manager.cc:5484] T 4c89a8bb1e4c499f9619676cc6af97d2 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.477296 7926 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.477584 7140 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.478480 9678 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.478921 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c89a8bb1e4c499f9619676cc6af97d2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.481725 7141 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.482291 9543 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.482836 10353 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.483485 10353 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.483896 7271 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.483958 10353 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.484232 10033 raft_consensus.cc:1050] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:08.484704 10353 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.488325 10033 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.494104 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c89a8bb1e4c499f9619676cc6af97d2 with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:08.495390 7796 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.504837 7927 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.505764 7534 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.505899 10107 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.506139 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.506728 10107 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.513232 7405 raft_consensus.cc:2932] T 4c89a8bb1e4c499f9619676cc6af97d2 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.526948 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.527654 10106 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.530652 10357 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.531857 7926 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.532564 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.533702 7141 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.538307 7929 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.540539 7000 catalog_manager.cc:5484] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.541250 7798 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.542876 7536 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.543501 10359 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.546658 7535 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.553916 7666 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.553923 9543 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.555759 7798 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.557914 7666 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.557852 9543 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.558526 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.562525 7405 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.565659 7798 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.567515 9392 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.574702 7405 consensus_queue.cc:237] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.578742 10361 tablet_replica.cc:330] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:08.579298 10361 raft_consensus.cc:2229] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:08.579488 7535 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.579800 10361 raft_consensus.cc:2258] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:08.580047 10353 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.581283 7926 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.581813 10353 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.581998 7140 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.582628 10353 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.584471 10074 raft_consensus.cc:1050] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:08.585686 10074 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:08.586148 10106 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.586150 10017 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.586851 10106 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.587649 10106 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.588275 10106 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.590595 7667 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.590847 7272 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.595337 9677 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.594877 7140 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.596802 6992 catalog_manager.cc:5484] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.598603 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.602996 7536 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.603129 7271 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.603705 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.604682 9544 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.606312 7926 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.601955 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.609544 9729 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.609864 7405 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.609872 10082 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.610641 10322 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.610905 7139 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.611485 10322 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.611819 9678 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.614315 10322 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.619727 10346 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.623176 9543 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.625026 10016 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.626636 7271 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.634797 7534 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.641872 7405 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.648450 10016 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.650122 10353 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.652352 7926 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.658007 7535 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.660113 6995 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.662146 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d55e6e4ce62d43838354b96c25b75086 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.661841 6992 catalog_manager.cc:5484] T d55e6e4ce62d43838354b96c25b75086 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.664963 7663 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.666260 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet d55e6e4ce62d43838354b96c25b75086 with delete_type TABLET_DATA_TOMBSTONED (TS c8f737f3f6994db297b47fb23dad1b13 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33072
I20240501 14:03:08.666965 9729 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.667335 7139 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.693620 10322 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.698782 7272 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.700393 7663 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.700919 9678 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.710043 6999 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.714234 7665 raft_consensus.cc:1261] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.716751 7534 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.736253 10140 raft_consensus.cc:1050] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:08.738090 10140 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.743052 9543 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.744004 7405 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.744160 7796 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.744736 7665 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.744763 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.745268 7926 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.745705 7141 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.746527 7273 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.755230 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20240501 14:03:08.755741 10326 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.756033 10361 ts_tablet_manager.cc:1875] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:08.756678 10326 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.758971 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.761325 7273 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.769606 10391 tablet_replica.cc:330] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: stopping tablet replica
I20240501 14:03:08.769937 10391 raft_consensus.cc:2229] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:08.770406 10391 raft_consensus.cc:2258] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:03:08.771656 7603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.201:33735, user_credentials={real_user=slave}} blocked reactor thread for 65324.8us
I20240501 14:03:08.771626 7797 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.778048 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.778249 9543 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.781487 7926 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.782418 7141 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.790892 10391 ts_tablet_manager.cc:1875] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:08.793414 10402 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:03:08.794620 7470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:33993, user_credentials={real_user=slave}} blocked reactor thread for 46005.7us
W20240501 14:03:08.796227 7469 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.200:43849, user_credentials={real_user=slave}} blocked reactor thread for 48373.3us
W20240501 14:03:08.796613 10398 log.cc:924] Time spent T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240501 14:03:08.796182 7796 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.798502 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20240501 14:03:08.802273 7471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 54542us
I20240501 14:03:08.823783 7666 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.825940 7402 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.826942 7534 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.833395 10374 consensus_queue.cc:1035] T d55e6e4ce62d43838354b96c25b75086 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.839143 7927 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.840637 7665 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.846877 7271 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.849486 7665 raft_consensus.cc:2932] T d55e6e4ce62d43838354b96c25b75086 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.850685 7796 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.850793 9544 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.851460 7929 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.852162 10347 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.852537 10334 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.855099 7405 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.855139 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.855437 7141 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.856850 10334 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.857228 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.859126 10107 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.862262 7929 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.864449 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 253a0abd26814fb29f9adede664c5988 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.865572 9544 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.872866 6992 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:08.876547 7141 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.878657 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 253a0abd26814fb29f9adede664c5988 with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:08.894344 10326 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.899694 7798 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:08.892753 7404 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.899246 7797 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.902189 7796 consensus_queue.cc:237] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:08.903098 7929 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.905476 7926 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.905599 9814 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.906152 10322 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.906723 7404 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.906728 10082 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.907300 10322 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.911659 10322 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.912775 7404 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.913005 7272 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.913776 9544 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.917107 7405 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.918699 9678 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.919332 7000 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.920229 7140 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.921962 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.930725 7140 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.930820 6998 catalog_manager.cc:5484] T f920ec86006647cc8e4efbf88dabd6cc P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.931289 10082 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.939181 7141 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.939795 7664 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.940625 10082 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.940732 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet f920ec86006647cc8e4efbf88dabd6cc with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:08.943645 7666 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.946591 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f920ec86006647cc8e4efbf88dabd6cc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.948226 7405 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.948752 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.948899 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.949550 7271 raft_consensus.cc:1261] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.949618 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.949246 9814 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.950696 10074 consensus_queue.cc:1035] T f920ec86006647cc8e4efbf88dabd6cc P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.951020 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.951563 10074 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.954980 7665 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.954509 7272 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.955667 9729 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.957454 9729 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.961570 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64b9b1b64def483aaed2a03f7d356915 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:08.961248 7664 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.962767 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.963156 9678 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.963711 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.967412 7533 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:08.971087 7535 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.970875 6992 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.972632 7140 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.973693 9729 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.974555 9543 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.976686 7405 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.976064 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.977033 7798 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.977263 10400 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.977401 7273 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.977836 10400 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.979094 10140 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.980753 10401 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.980463 7927 raft_consensus.cc:2932] T f920ec86006647cc8e4efbf88dabd6cc P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:08.982153 7929 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:08.982714 10401 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:08.985693 10326 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.987210 7926 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.989171 9544 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.993037 7798 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.994623 7403 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:08.995314 6999 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:08.998320 7401 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:09.001989 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f56b1674c3ec4bdc9c2bd82a231d0112 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:09.002377 7139 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.003059 10391 ts_tablet_manager.cc:1888] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.003304 10391 log.cc:1196] T d55e6e4ce62d43838354b96c25b75086 P c8f737f3f6994db297b47fb23dad1b13: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/d55e6e4ce62d43838354b96c25b75086
I20240501 14:03:09.004035 7516 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b9b1b64def483aaed2a03f7d356915 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:09.003998 7667 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.004379 10033 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.006596 7535 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:09.008355 6984 catalog_manager.cc:4830] TS c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): tablet d55e6e4ce62d43838354b96c25b75086 (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
I20240501 14:03:09.012100 7273 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:09.018669 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:09.019805 10361 ts_tablet_manager.cc:1888] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.020073 10361 log.cc:1196] T 4c89a8bb1e4c499f9619676cc6af97d2 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/4c89a8bb1e4c499f9619676cc6af97d2
I20240501 14:03:09.021348 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet 4c89a8bb1e4c499f9619676cc6af97d2 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:09.021904 10361 tablet_replica.cc:330] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:09.024219 10361 raft_consensus.cc:2229] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.024699 10361 raft_consensus.cc:2258] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.026610 10401 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.030056 7141 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:09.032380 10361 ts_tablet_manager.cc:1875] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.048555 10415 tablet_replica.cc:330] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:09.049060 10415 raft_consensus.cc:2229] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.049410 10415 raft_consensus.cc:2258] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.051656 10415 ts_tablet_manager.cc:1875] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.053977 10361 ts_tablet_manager.cc:1888] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.054240 10361 log.cc:1196] T 253a0abd26814fb29f9adede664c5988 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/253a0abd26814fb29f9adede664c5988
I20240501 14:03:09.055245 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet 253a0abd26814fb29f9adede664c5988 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:09.056505 10361 tablet_replica.cc:330] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:09.056753 10361 raft_consensus.cc:2229] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.057191 10361 raft_consensus.cc:2258] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.058243 10361 ts_tablet_manager.cc:1875] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.062615 10415 ts_tablet_manager.cc:1888] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.062844 10415 log.cc:1196] T f920ec86006647cc8e4efbf88dabd6cc P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/f920ec86006647cc8e4efbf88dabd6cc
I20240501 14:03:09.064074 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet f920ec86006647cc8e4efbf88dabd6cc (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:09.064836 10415 tablet_replica.cc:330] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:09.065106 10415 raft_consensus.cc:2229] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.065493 10415 raft_consensus.cc:2258] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.066607 10415 ts_tablet_manager.cc:1875] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.067018 10361 ts_tablet_manager.cc:1888] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.067260 10361 log.cc:1196] T f56b1674c3ec4bdc9c2bd82a231d0112 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/f56b1674c3ec4bdc9c2bd82a231d0112
I20240501 14:03:09.068089 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet f56b1674c3ec4bdc9c2bd82a231d0112 (table rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]) successfully deleted
I20240501 14:03:09.072911 10415 ts_tablet_manager.cc:1888] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.073130 10415 log.cc:1196] T 64b9b1b64def483aaed2a03f7d356915 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:09.073963 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 64b9b1b64def483aaed2a03f7d356915 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:09.140079 7405 consensus_queue.cc:237] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:09.142488 9678 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.142637 7929 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.142673 7272 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.142908 7666 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.143122 10374 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.143442 10353 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.143764 10357 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.144037 7798 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.144170 7140 raft_consensus.cc:1261] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.144354 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.144876 10016 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.145545 10017 consensus_queue.cc:1035] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.148053 10017 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.148963 7272 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.150013 9678 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.150641 7798 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.155057 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11c87a132b49460d89c3122c3c4f2c32 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:09.157214 6995 catalog_manager.cc:5484] T 11c87a132b49460d89c3122c3c4f2c32 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.157464 7929 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.158298 7666 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.154924 7140 raft_consensus.cc:2932] T 11c87a132b49460d89c3122c3c4f2c32 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:09.162376 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c87a132b49460d89c3122c3c4f2c32 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:09.164917 10415 tablet_replica.cc:330] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:09.165441 10415 raft_consensus.cc:2229] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.166334 10415 raft_consensus.cc:2258] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.167483 10415 ts_tablet_manager.cc:1875] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.172495 10415 ts_tablet_manager.cc:1888] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.172602 10415 log.cc:1196] T 11c87a132b49460d89c3122c3c4f2c32 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/11c87a132b49460d89c3122c3c4f2c32
I20240501 14:03:09.173314 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 11c87a132b49460d89c3122c3c4f2c32 (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:09.224542 7140 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:09.227953 7929 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.228055 7405 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.228096 7798 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.228586 10396 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.229378 10424 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:03:09.230537 7272 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.231287 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.232592 7533 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.232838 10396 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.233155 10425 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.235065 7666 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.237404 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.238663 10132 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.239259 7405 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.245455 6992 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.246994 7798 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.248855 7533 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.249444 7929 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.255457 7272 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.255781 7140 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:09.269641 7535 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.271210 7929 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.272166 10132 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.274008 10132 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.274444 7798 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:09.287405 7666 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.288426 7796 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.288730 7272 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.289683 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.292479 10132 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.292822 10424 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.297003 7666 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
W20240501 14:03:09.299333 7074 consensus_peers.cc:485] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11da808a7f774e5ea7bd7ef298f1180e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:09.300009 10437 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.300853 7405 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.301865 7272 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.305555 10132 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.307456 7405 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.308928 7666 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.310792 6986 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 11da808a7f774e5ea7bd7ef298f1180e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:09.313179 7535 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.313531 7929 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.316738 6993 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.306968 7798 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.612735 7926 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:09.616189 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.616675 7798 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.616716 7405 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.617313 7535 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.617444 7666 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.617601 7272 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.617792 9914 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.618162 9731 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.618592 10479 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.618980 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.619503 10480 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.623979 9731 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.638108 9731 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.638738 7533 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.645602 6993 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.645658 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.647322 7798 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.647609 7405 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.647956 7666 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.654234 7926 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:09.656682 7140 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.662910 7271 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.663699 7798 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.664434 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.665215 10480 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.665299 7533 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.668334 10480 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.670028 7664 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.674767 9914 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.677304 9914 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.678776 7664 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.681830 7533 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.684356 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9812475b87e9492d8c9450b77d34ea6b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:09.685257 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.685729 7405 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.686084 10479 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.686750 9731 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.687428 6998 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.688556 7141 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.688324 7405 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.697881 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.698151 7926 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
W20240501 14:03:09.699628 7865 consensus_peers.cc:485] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9812475b87e9492d8c9450b77d34ea6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:09.700476 7796 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:09.701234 10495 ts_tablet_manager.cc:902] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:09.713351 7405 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.713531 7140 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.714176 7533 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.714524 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.714681 7272 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.714839 10491 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.715135 9731 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.715647 7796 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.716156 10480 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.716452 7664 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.716461 9731 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.718721 10480 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.727856 9731 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.729985 7141 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.730578 7798 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.732503 7405 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.733232 7272 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.738476 6998 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.738538 7533 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.748715 10495 tablet_copy_client.cc:308] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:09.749017 7926 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:09.752660 7535 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.754359 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.756758 7405 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.757511 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.758685 7272 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.759263 7140 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.759784 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.760136 7664 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.760967 7798 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.762195 10491 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.762780 10491 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.763690 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:03:09.764649 7862 consensus_peers.cc:485] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd9f7d11351b4f8d84021720488abf25. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:09.765514 9731 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.766052 7666 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:09.766095 7535 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.773310 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd9f7d11351b4f8d84021720488abf25 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:09.772868 7796 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.773413 7272 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.778959 7141 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.784147 6999 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.786413 7405 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.794775 7666 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.798821 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet 11da808a7f774e5ea7bd7ef298f1180e from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:48843)
I20240501 14:03:09.799142 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet 11da808a7f774e5ea7bd7ef298f1180e from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:48843: session id = 1c61c125cd144590ae1e54a2f87d9187-11da808a7f774e5ea7bd7ef298f1180e
I20240501 14:03:09.803030 7161 tablet_copy_source_session.cc:205] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:09.806107 10495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:09.809027 7926 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:09.810611 10495 tablet_copy_client.cc:783] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:09.810868 10495 tablet_copy_client.cc:647] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:09.812199 7141 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.812569 7405 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.812834 7535 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.813159 10491 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.813656 7272 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.813608 9914 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.813822 10495 tablet_copy_client.cc:523] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:09.814013 7796 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.813992 7666 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.815928 9914 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.816227 10491 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.816612 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.817216 10479 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.818739 10495 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:09.823556 10491 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.824656 7796 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.826854 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.830945 6992 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.840200 7405 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.841277 7535 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.842761 7141 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.843137 7666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.846915 7926 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
W20240501 14:03:09.852425 7862 consensus_peers.cc:485] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab7d2ddba3440f0a1311f9eda7d3930. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:09.854673 7140 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.854709 7796 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.855357 9731 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.855942 7271 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.856595 9731 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.856809 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.859208 7666 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.860903 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.863742 10478 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.864492 7405 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.865134 7535 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:09.865095 10491 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.865846 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.866544 7405 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.868057 9914 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:09.871634 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:09.873698 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.872776 6995 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.874871 7796 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.875979 7535 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.888374 7666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:09.911170 7140 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:09.914113 7926 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.914857 10132 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.915395 7796 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.916900 10132 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.918735 7405 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.919158 7535 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.919318 10437 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.919859 10426 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.923766 7271 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.924749 10495 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:09.924898 10437 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.925443 10495 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:09.925788 10495 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.107s user 0.049s sys 0.001s
I20240501 14:03:09.926823 10495 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:09.927284 10495 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:09.927450 7666 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:09.927649 10495 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:09.928062 10132 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.939168 10495 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20240501 14:03:09.941164 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-11da808a7f774e5ea7bd7ef298f1180e received from {username='slave'} at 127.22.5.202:48843
I20240501 14:03:09.941362 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-11da808a7f774e5ea7bd7ef298f1180e on tablet 11da808a7f774e5ea7bd7ef298f1180e with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:09.964046 10437 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.966779 7272 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.978577 6998 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.982112 7926 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.995275 7405 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:09.997007 7141 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.003325 7405 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.004225 10437 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.004473 7271 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.005241 10437 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.008273 7798 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.009351 10132 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:09.997536 7666 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:10.010789 7798 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:10.011678 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.012466 10437 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.016000 10437 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.021014 7926 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.031584 7664 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.033411 10426 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.035842 7664 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.036515 7533 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.037458 10132 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.037573 6986 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2264386e494644aa84e8109c816aaefa with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:10.039136 7533 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:10.043658 7405 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.045913 7533 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.049269 7271 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.050148 6998 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.064177 7798 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
W20240501 14:03:10.068149 7077 consensus_peers.cc:485] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2264386e494644aa84e8109c816aaefa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:10.086371 7141 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:10.089800 7405 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.089965 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.090586 7798 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.090670 10437 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.090842 7926 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.091223 10437 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.092087 10426 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.094553 10426 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.104971 10437 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.106027 7664 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.107012 7533 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:10.107503 10132 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.107821 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:10.108392 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.110781 7405 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.112435 6998 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:10.113950 7926 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.114032 7533 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.115247 7664 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.119171 7798 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:10.149780 7141 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
W20240501 14:03:10.154596 7077 consensus_peers.cc:485] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b227b60efb7343bba8c4e89f81c3f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:10.155604 7535 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.156183 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.156937 7664 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.157618 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.160271 7796 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.160627 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.161971 7926 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.162179 7401 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:10.162303 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.162783 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.162978 10132 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.163419 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:10.166565 10541 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.167272 7796 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.168601 7664 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.171119 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.172147 6986 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b227b60efb7343bba8c4e89f81c3f6ff with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:10.173956 7926 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.178328 7535 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.185443 6999 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.197983 7401 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:10.200760 10556 ts_tablet_manager.cc:902] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:10.216281 10556 tablet_copy_client.cc:308] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:10.232403 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 9812475b87e9492d8c9450b77d34ea6b from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:33315)
I20240501 14:03:10.232750 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 9812475b87e9492d8c9450b77d34ea6b from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:33315: session id = 41ab226fa6b7403b8b44642e9470517d-9812475b87e9492d8c9450b77d34ea6b
I20240501 14:03:10.237006 7949 tablet_copy_source_session.cc:205] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.239410 10556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.247660 10556 tablet_copy_client.cc:783] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.248096 10556 tablet_copy_client.cc:647] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.251999 10556 tablet_copy_client.cc:523] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.259043 10556 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:10.304792 10495 ts_tablet_manager.cc:902] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:10.314637 9814 raft_consensus.cc:1203] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:10.320302 10495 tablet_copy_client.cc:308] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:10.338449 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet bd9f7d11351b4f8d84021720488abf25 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:60515)
I20240501 14:03:10.338635 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet bd9f7d11351b4f8d84021720488abf25 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:60515: session id = 1c61c125cd144590ae1e54a2f87d9187-bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:10.342185 7949 tablet_copy_source_session.cc:205] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.343704 10495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.353968 10495 tablet_copy_client.cc:783] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.354938 10495 tablet_copy_client.cc:647] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.357157 10495 tablet_copy_client.cc:523] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.363176 10495 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:10.385807 10556 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.386353 10556 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:10.386698 10556 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.128s user 0.081s sys 0.000s
I20240501 14:03:10.387701 10556 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.388114 10556 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:10.388376 10556 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.404589 10556 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.018s user 0.008s sys 0.001s
I20240501 14:03:10.409742 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-9812475b87e9492d8c9450b77d34ea6b received from {username='slave'} at 127.22.5.201:33315
I20240501 14:03:10.409934 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-9812475b87e9492d8c9450b77d34ea6b on tablet 9812475b87e9492d8c9450b77d34ea6b with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:10.453428 10595 ts_tablet_manager.cc:902] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:10.469066 10556 ts_tablet_manager.cc:902] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:10.485033 10595 tablet_copy_client.cc:308] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:10.487370 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:60515)
I20240501 14:03:10.487524 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:60515: session id = 1c61c125cd144590ae1e54a2f87d9187-1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:10.490495 7949 tablet_copy_source_session.cc:205] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.493002 10595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.499831 10595 tablet_copy_client.cc:783] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.500201 10595 tablet_copy_client.cc:647] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.502282 10595 tablet_copy_client.cc:523] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.506538 10595 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:10.508257 10556 tablet_copy_client.cc:308] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:10.510169 10495 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.510614 10495 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:10.510916 10495 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.148s user 0.059s sys 0.013s
I20240501 14:03:10.511801 10495 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:10.512111 10495 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:10.512362 10495 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:10.513041 10495 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:10.513839 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-bd9f7d11351b4f8d84021720488abf25 received from {username='slave'} at 127.22.5.202:60515
I20240501 14:03:10.514008 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-bd9f7d11351b4f8d84021720488abf25 on tablet bd9f7d11351b4f8d84021720488abf25 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:10.519490 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet 2264386e494644aa84e8109c816aaefa from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:43217)
I20240501 14:03:10.519706 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet 2264386e494644aa84e8109c816aaefa from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:43217: session id = 41ab226fa6b7403b8b44642e9470517d-2264386e494644aa84e8109c816aaefa
I20240501 14:03:10.523739 7161 tablet_copy_source_session.cc:205] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.525527 10556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.531265 10556 tablet_copy_client.cc:783] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.531505 10556 tablet_copy_client.cc:647] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.533361 10556 tablet_copy_client.cc:523] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.535842 10556 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:10.573616 10556 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.574026 10556 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:10.574301 10556 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.039s user 0.030s sys 0.008s
I20240501 14:03:10.575183 10556 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.575471 10556 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:10.575744 10556 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.576395 10556 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:03:10.577463 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-2264386e494644aa84e8109c816aaefa received from {username='slave'} at 127.22.5.201:43217
I20240501 14:03:10.577598 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-2264386e494644aa84e8109c816aaefa on tablet 2264386e494644aa84e8109c816aaefa with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:10.586699 10595 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.587177 10595 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:10.587495 10595 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.081s user 0.052s sys 0.013s
I20240501 14:03:10.588184 10595 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:10.588447 10595 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:10.588654 10595 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:10.589241 10595 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:03:10.589905 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-1ab7d2ddba3440f0a1311f9eda7d3930 received from {username='slave'} at 127.22.5.202:60515
I20240501 14:03:10.590044 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-1ab7d2ddba3440f0a1311f9eda7d3930 on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:10.695729 9678 raft_consensus.cc:1203] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:10.731029 10556 ts_tablet_manager.cc:902] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:10.732278 10556 tablet_copy_client.cc:308] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:10.732968 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:43217)
I20240501 14:03:10.733166 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:43217: session id = 41ab226fa6b7403b8b44642e9470517d-b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:10.735249 7161 tablet_copy_source_session.cc:205] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.736519 10556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.741430 10556 tablet_copy_client.cc:783] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.741685 10556 tablet_copy_client.cc:647] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.743331 10556 tablet_copy_client.cc:523] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.746074 10556 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:10.775252 9814 raft_consensus.cc:1203] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:10.781247 10556 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.781723 10556 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:10.781991 10556 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20240501 14:03:10.782637 10556 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.782881 10556 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:10.783092 10556 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:10.783562 10556 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:03:10.784298 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-b227b60efb7343bba8c4e89f81c3f6ff received from {username='slave'} at 127.22.5.201:43217
I20240501 14:03:10.784471 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-b227b60efb7343bba8c4e89f81c3f6ff on tablet b227b60efb7343bba8c4e89f81c3f6ff with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:10.797024 10541 raft_consensus.cc:1050] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:10.798007 10541 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:10.801365 7926 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.801599 7401 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.802230 7272 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.802176 10643 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.804102 7666 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.804414 10643 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.804904 7535 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.805073 10643 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.805683 7798 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.805768 9814 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:10.806875 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.808972 10554 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.809276 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.809742 10554 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:10.814774 10554 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.818208 7272 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.823830 6999 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.823755 7926 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.826485 7535 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.827785 7401 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.830694 9814 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.838953 7664 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.850378 7798 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.863943 7140 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:10.867755 9814 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.867951 9678 raft_consensus.cc:1203] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:10.868970 7929 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.869050 7535 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.869545 10541 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.869822 7272 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.871037 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.871753 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.872206 10426 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.874545 7401 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.875177 10541 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.877707 10426 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.878273 7272 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.881687 7798 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.882380 10554 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.883137 9814 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.884428 6995 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.884455 7405 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.884881 7798 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.892165 6986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11da808a7f774e5ea7bd7ef298f1180e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:10.892320 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 11da808a7f774e5ea7bd7ef298f1180e with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:10.893749 7535 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.893273 7926 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:10.902472 10656 tablet_replica.cc:330] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:10.902693 10656 raft_consensus.cc:2229] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:10.902920 10656 raft_consensus.cc:2258] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:10.904871 10656 ts_tablet_manager.cc:1875] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:10.916646 10656 ts_tablet_manager.cc:1888] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:10.916921 10656 log.cc:1196] T 11da808a7f774e5ea7bd7ef298f1180e P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/11da808a7f774e5ea7bd7ef298f1180e
I20240501 14:03:10.918113 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet 11da808a7f774e5ea7bd7ef298f1180e (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:11.036751 9814 raft_consensus.cc:1203] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:11.150813 9914 raft_consensus.cc:1050] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:11.152256 9914 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.156039 7401 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.156167 9678 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.156896 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.156914 7272 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.157609 9731 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.157968 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.163004 7798 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.163587 7664 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.163677 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.164794 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.164923 7535 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.164712 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.165699 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.166427 9731 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.175383 7140 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.174566 10478 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.183036 6993 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.193006 7401 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.197749 7534 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.198575 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.198575 7664 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.206586 7798 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.208159 9678 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.214781 7926 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.218264 7272 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.218279 9678 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.218545 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.219058 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.219645 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.219835 7401 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.220175 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.220577 10696 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.223335 10688 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.223994 7140 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.225231 7401 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.228120 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9812475b87e9492d8c9450b77d34ea6b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:11.232728 7664 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.234239 7798 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.234369 10478 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.230885 6995 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.238389 10697 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.238101 7664 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.240700 7798 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.241313 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812475b87e9492d8c9450b77d34ea6b with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:11.244300 9678 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.249429 7271 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.254118 10702 tablet_replica.cc:330] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:11.254380 10702 raft_consensus.cc:2229] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.254667 10702 raft_consensus.cc:2258] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.255661 10702 ts_tablet_manager.cc:1875] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.263962 10702 ts_tablet_manager.cc:1888] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:11.264211 10702 log.cc:1196] T 9812475b87e9492d8c9450b77d34ea6b P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/9812475b87e9492d8c9450b77d34ea6b
I20240501 14:03:11.265197 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 9812475b87e9492d8c9450b77d34ea6b (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:11.270107 9678 raft_consensus.cc:1203] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:11.390108 10554 raft_consensus.cc:1050] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:11.391237 10554 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.394551 7535 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.394614 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.395159 10697 raft_consensus.cc:1050] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:11.395671 10554 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.396158 7926 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.396492 10697 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:11.396775 10554 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.399329 10554 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.400144 7271 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.400832 10554 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.400878 7664 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.401504 7798 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.401489 10541 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.402915 10541 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.403118 7401 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.403703 10541 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.420203 10554 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.425655 7664 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.427377 10688 raft_consensus.cc:1050] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:11.427037 7929 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.428479 10688 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:11.433629 9678 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.434463 6992 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.438081 9814 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.438150 7405 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.438102 7271 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.440613 7140 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.440657 7272 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.441401 10688 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.445693 7796 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.446789 7535 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.447819 7141 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.447876 10699 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.448333 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.448567 10699 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.449760 7405 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.450340 7272 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.450054 7535 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.451104 10696 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.452503 9914 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.453538 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.459100 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.459733 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.462272 7535 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.462862 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.467388 7797 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.473290 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.475013 9813 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.475625 7664 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.476034 10697 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.491012 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.491348 7664 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.497664 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:03:11.519357 10729 log.cc:924] Time spent T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240501 14:03:11.534483 7798 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.539624 7140 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.537477 7401 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.544318 7272 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.544956 10541 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.545918 7533 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.547930 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.549805 10643 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.550415 10643 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.555764 10541 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
W20240501 14:03:11.557399 7600 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:45423, user_credentials={real_user=slave}} blocked reactor thread for 114075us
I20240501 14:03:11.559619 10478 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.564412 7402 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.567101 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.568733 10643 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.568821 7797 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.571805 7273 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.574792 9677 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.558653 7533 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.579859 7404 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.580948 10426 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.583593 7795 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.584221 10426 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.586227 6993 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.588270 6986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2264386e494644aa84e8109c816aaefa with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:11.589239 7273 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.590430 6991 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:11.594051 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 2264386e494644aa84e8109c816aaefa with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:11.594884 10760 tablet_replica.cc:330] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:11.595213 10760 raft_consensus.cc:2229] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.595784 10760 raft_consensus.cc:2258] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.596866 10760 ts_tablet_manager.cc:1875] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.602283 10760 ts_tablet_manager.cc:1888] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:11.602412 10760 log.cc:1196] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/2264386e494644aa84e8109c816aaefa
I20240501 14:03:11.603124 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet 2264386e494644aa84e8109c816aaefa (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:11.605818 9914 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.607210 9814 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.608935 7141 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.609786 7929 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.610049 7666 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.613638 9813 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.617875 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.618170 7535 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.622431 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.622515 7273 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.627746 7666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.629283 7402 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.630173 6995 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.634877 7798 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.641866 7929 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:11.646112 7534 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.646937 10697 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.647267 7404 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.647455 7798 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.648017 10688 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.648481 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.649713 7141 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.650820 9914 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.650153 7273 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.652042 10697 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.652779 7401 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.653980 7140 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.656778 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd9f7d11351b4f8d84021720488abf25 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:11.656538 7929 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:11.659257 7796 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.661938 7666 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.662489 10688 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.663190 7271 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.663908 10697 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.663689 7533 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.664404 7139 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.665568 7404 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.666183 7798 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.666707 9914 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.667519 9731 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.670146 10696 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.670593 7797 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.669458 6998 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.671751 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.675158 7405 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.674903 7795 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.676026 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.677027 10643 raft_consensus.cc:1050] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:11.677774 7405 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.678278 10643 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.683153 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:11.685799 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.686091 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.688024 6991 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.688405 7273 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.694993 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:11.695411 10702 tablet_replica.cc:330] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:11.695645 10702 raft_consensus.cc:2229] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.696027 10702 raft_consensus.cc:2258] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.697085 10702 ts_tablet_manager.cc:1875] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.703369 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f7d11351b4f8d84021720488abf25 with delete_type TABLET_DATA_TOMBSTONED (TS db6b830d9bbe4451bd349c6ad6e14bab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35026
I20240501 14:03:11.703464 9811 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.686018 7666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.705158 9677 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.705874 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.710261 10478 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.712698 9811 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.713532 10702 ts_tablet_manager.cc:1888] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:11.713670 10702 log.cc:1196] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:11.714452 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:11.714928 10760 tablet_replica.cc:330] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: stopping tablet replica
I20240501 14:03:11.715219 10760 raft_consensus.cc:2229] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.715866 10760 raft_consensus.cc:2258] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.716660 7666 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:11.717860 10760 ts_tablet_manager.cc:1875] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.718619 7402 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.719172 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.723547 7929 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.724995 7533 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.725482 7405 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.724510 7141 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.726271 10737 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.727274 10737 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.729548 10071 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.730684 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.736507 7796 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.737502 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.739760 10760 ts_tablet_manager.cc:1888] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:11.739907 10760 log.cc:1196] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-4/wal/wals/bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:11.744527 6984 catalog_manager.cc:4830] TS db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): tablet bd9f7d11351b4f8d84021720488abf25 (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:11.748744 7926 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.749456 7796 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.749557 10727 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.750327 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.750423 10727 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.750963 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.751717 10740 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.758709 7533 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.759316 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.759960 7534 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.765156 7141 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.751714 7666 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.767329 6992 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:11.771625 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.767558 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.777752 7664 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:11.782735 7534 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.782711 10541 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.783435 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:11.785521 7666 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.756798 7273 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:11.791155 7271 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.792089 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.793854 7929 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.793630 6997 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.794523 7272 raft_consensus.cc:858] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
W20240501 14:03:11.794663 7272 raft_consensus.cc:1565] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 5' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:11.795632 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:11.803892 7272 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.806813 7797 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.807554 10740 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.808869 7797 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.809422 7795 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.809484 9678 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
W20240501 14:03:11.816576 7600 consensus_peers.cc:485] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e104d44c5fa4a29be94a33cf7e93ba6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:11.818637 10071 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.821270 9812 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.822182 10478 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.827700 7534 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.829161 7929 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.829697 9812 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:11.833084 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:11.836336 7796 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.836469 6992 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:11.837495 7273 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.838348 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.843752 7139 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.846334 10769 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:11.846549 7401 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:11.847272 10071 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:11.848771 7401 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:11.852454 7401 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.858796 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.863481 7402 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.879963 7140 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:11.888136 7534 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.892975 7140 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:11.897567 7402 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.898042 7666 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.898972 7929 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.900687 7796 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.900772 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.901703 10790 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.901961 9677 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.901963 10541 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.902163 10643 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.903844 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:11.905756 10790 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.906939 10790 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.907796 7929 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.911355 6986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b227b60efb7343bba8c4e89f81c3f6ff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:11.912164 10643 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:11.913789 9677 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.914247 7796 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.918534 6998 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:11.920504 7666 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.925349 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:11.925743 10702 tablet_replica.cc:330] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:11.925976 10702 raft_consensus.cc:2229] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.926420 10702 raft_consensus.cc:2258] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.927996 10702 ts_tablet_manager.cc:1875] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.926606 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.936604 7402 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:11.964919 10702 ts_tablet_manager.cc:1888] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:11.965209 10702 log.cc:1196] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:11.965991 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet b227b60efb7343bba8c4e89f81c3f6ff (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:12.237355 7402 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:12.240991 7140 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.241376 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.241354 7926 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.242019 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.242655 10661 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.243456 10661 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.253031 7534 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.254181 10661 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.254362 7273 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.255373 7666 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.256485 10661 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.257519 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.269647 10553 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.275313 7926 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.275321 7796 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.279050 7273 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.279047 7664 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.281198 7534 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.283162 6998 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:12.291123 7139 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.294116 7401 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
W20240501 14:03:12.302999 7338 consensus_peers.cc:485] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cf39e6be52426c93c1534754be7264. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:12.304710 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.304811 7271 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.305655 7926 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.305600 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.306197 7139 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.306241 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.306669 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.307909 10553 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.309106 7664 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.310629 7534 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.311497 10827 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.311921 10827 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.315367 10661 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.317145 7271 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.318977 7797 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.320081 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 60cf39e6be52426c93c1534754be7264 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:12.322317 7664 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.327098 6997 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.327486 7926 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.344087 7139 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.344060 7535 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.344844 7929 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:12.349876 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.350005 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.350610 10478 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.351188 7139 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.351740 10478 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.352289 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.352257 10478 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.354218 7534 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.354265 9914 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.354851 10478 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.355345 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:12.360324 9914 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.390195 10478 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.391183 7534 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.396522 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.398430 6998 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.400429 10843 ts_tablet_manager.cc:902] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:12.402642 7273 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.406774 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.416815 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.428388 7929 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:12.433876 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.434204 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.434571 7534 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.435343 10697 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.453801 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.455000 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.456796 10849 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.457278 10697 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.458067 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.458917 7139 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:12.459656 10849 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:12.460243 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:12.460884 7139 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:12.466033 10688 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.476132 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2dc517698d5a4ae19525c96334ce7417 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:12.477497 7534 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.477654 7139 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.479584 6991 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:12.484906 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.484882 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:12.486413 7601 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:33993, user_credentials={real_user=slave}} blocked reactor thread for 76445.7us
I20240501 14:03:12.497042 7271 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.516656 7402 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:12.529068 10843 tablet_copy_client.cc:308] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:42421
W20240501 14:03:12.530287 7862 consensus_peers.cc:485] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dc517698d5a4ae19525c96334ce7417. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:12.552453 7687 tablet_copy_service.cc:140] P db6b830d9bbe4451bd349c6ad6e14bab: Received BeginTabletCopySession request for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:44945)
I20240501 14:03:12.552639 7687 tablet_copy_service.cc:161] P db6b830d9bbe4451bd349c6ad6e14bab: Beginning new tablet copy session on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:44945: session id = 1c61c125cd144590ae1e54a2f87d9187-4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:12.555634 7687 tablet_copy_source_session.cc:205] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:12.557006 10843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:12.562012 10843 tablet_copy_client.cc:783] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:12.562271 10843 tablet_copy_client.cc:647] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:12.564105 10843 tablet_copy_client.cc:523] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:12.568512 10843 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:12.616509 10843 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:12.616884 10843 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:12.617152 10843 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.049s user 0.044s sys 0.000s
I20240501 14:03:12.617758 10843 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:12.618003 10843 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:12.618155 10843 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:12.618856 10843 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:12.619666 7687 tablet_copy_service.cc:342] P db6b830d9bbe4451bd349c6ad6e14bab: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-4e104d44c5fa4a29be94a33cf7e93ba6 received from {username='slave'} at 127.22.5.202:44945
I20240501 14:03:12.619797 7687 tablet_copy_service.cc:433] P db6b830d9bbe4451bd349c6ad6e14bab: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-4e104d44c5fa4a29be94a33cf7e93ba6 on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:12.792943 9813 raft_consensus.cc:1203] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:12.794075 10876 ts_tablet_manager.cc:902] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:12.797691 10876 tablet_copy_client.cc:308] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:12.798410 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 60cf39e6be52426c93c1534754be7264 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:33293)
I20240501 14:03:12.798578 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 60cf39e6be52426c93c1534754be7264 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:33293: session id = cc3a47f051a546a5a7788d088f061dd5-60cf39e6be52426c93c1534754be7264
I20240501 14:03:12.801335 7425 tablet_copy_source_session.cc:205] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:12.802933 10876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:12.808472 10876 tablet_copy_client.cc:783] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:12.808724 10876 tablet_copy_client.cc:647] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:12.810737 10876 tablet_copy_client.cc:523] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:12.814052 10876 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:12.858644 10876 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:12.859179 10876 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:12.859467 10876 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.046s user 0.033s sys 0.008s
I20240501 14:03:12.860117 10876 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:12.860383 10876 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:12.860606 10876 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:12.861366 10876 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 14:03:12.862481 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-60cf39e6be52426c93c1534754be7264 received from {username='slave'} at 127.22.5.200:33293
I20240501 14:03:12.862591 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-60cf39e6be52426c93c1534754be7264 on tablet 60cf39e6be52426c93c1534754be7264 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:12.874239 10876 ts_tablet_manager.cc:902] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:12.875540 10876 tablet_copy_client.cc:308] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:12.881853 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 2dc517698d5a4ae19525c96334ce7417 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:57433)
I20240501 14:03:12.882016 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 2dc517698d5a4ae19525c96334ce7417 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:57433: session id = cc3a47f051a546a5a7788d088f061dd5-2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:12.884253 7949 tablet_copy_source_session.cc:205] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:12.885635 10876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:12.890116 10876 tablet_copy_client.cc:783] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:12.890363 10876 tablet_copy_client.cc:647] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:12.891922 10876 tablet_copy_client.cc:523] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:12.894229 10876 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:12.940816 10876 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:12.941367 10876 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:12.941663 10876 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.048s user 0.043s sys 0.003s
I20240501 14:03:12.942546 10876 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:12.943480 10876 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:12.943722 10876 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:12.944346 10876 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:12.945101 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-2dc517698d5a4ae19525c96334ce7417 received from {username='slave'} at 127.22.5.200:57433
I20240501 14:03:12.945279 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-2dc517698d5a4ae19525c96334ce7417 on tablet 2dc517698d5a4ae19525c96334ce7417 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:13.250705 10854 raft_consensus.cc:1050] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:13.251854 10854 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:13.254981 7139 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.255388 7929 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.255575 7402 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.255667 10071 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.256397 7271 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.257236 9812 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.257349 10921 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.259385 10845 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.259603 7797 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.260129 10845 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.260469 10921 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.261870 7534 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.262598 10845 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.263549 10071 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.286574 10071 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.292672 7796 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.298846 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.301561 7140 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.309171 7271 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.311728 9813 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.315634 6998 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:13.317287 7534 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.324110 7402 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.341269 9543 raft_consensus.cc:1203] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:13.342067 7666 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:13.356899 9813 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.357431 7140 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.357565 10931 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.357666 7402 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.358055 10071 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.358294 10845 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.360983 7929 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.361586 10931 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.362653 7271 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.363186 10845 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.364005 10854 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.365085 7797 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.364862 7402 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.366765 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.368347 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:13.368749 10921 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.368412 7273 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.370397 7140 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.374691 9813 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.375510 7797 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.375918 7000 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:13.382134 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:13.388567 10943 tablet_replica.cc:330] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:13.388911 10943 raft_consensus.cc:2229] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:13.389699 10943 raft_consensus.cc:2258] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:13.390691 10943 ts_tablet_manager.cc:1875] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:13.398483 10943 ts_tablet_manager.cc:1888] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:13.398948 10943 log.cc:1196] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:13.400074 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:13.483980 9543 raft_consensus.cc:1203] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:13.634608 7797 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:13.637523 7929 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.639092 7664 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.640080 7534 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.639822 10959 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.641003 7139 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.641057 10961 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.642074 7402 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.643126 10791 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.643498 10961 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.644258 10791 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.658211 7273 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:13.659166 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:13.667546 10961 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.668377 7401 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.668953 7929 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.669530 7139 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.671275 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.676021 6999 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.675962 7534 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.677691 7273 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:13.691943 7797 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:13.696058 7534 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.697957 10960 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.704113 7139 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.705406 10960 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.706522 7929 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.707113 10974 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.713146 7664 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.712857 10960 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.714442 7401 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.714883 7139 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.718019 7273 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:13.718451 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.718952 10959 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.719194 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:13.720890 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d2a7bfaac904352a9c9c12c09b563f5 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:13.721230 7534 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.722458 7929 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.724079 7000 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.729488 7401 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.731774 7273 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:13.738108 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:13.745604 7731 consensus_peers.cc:485] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d2a7bfaac904352a9c9c12c09b563f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:13.876964 10849 raft_consensus.cc:1050] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:13.877949 10849 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:13.881419 7139 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.882048 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.882313 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.882398 7534 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.882973 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.883096 9543 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.883515 7401 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.883734 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.883975 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.884268 10919 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.884778 10919 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.888129 10919 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.888172 7273 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.888739 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.902668 10688 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.904261 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.904695 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.906914 10940 raft_consensus.cc:1050] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:13.907981 10940 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:13.907860 7534 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.909989 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.912302 7139 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.914644 6997 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:13.912286 9543 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.934566 7926 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:13.937821 7140 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.937916 7141 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.938570 7271 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.938611 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.939846 10828 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.940019 9543 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.940490 10828 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.942485 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.942853 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.943399 10688 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.945915 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.946820 10988 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.930847 7273 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.950349 10988 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.951789 7666 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.953799 7141 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.958688 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dc517698d5a4ae19525c96334ce7417 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:13.961669 9543 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.963387 6999 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:13.966279 7796 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.972039 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.973923 10988 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.975703 7272 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:13.977298 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc517698d5a4ae19525c96334ce7417 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
W20240501 14:03:13.989460 7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:41451, user_credentials={real_user=slave}} blocked reactor thread for 49744.6us
I20240501 14:03:13.993539 7926 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.994619 9543 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.994561 10828 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:13.995292 7401 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.997032 10997 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.997642 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.998962 7534 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:13.999435 11002 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:14.000205 11002 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:14.000434 7666 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:14.001096 11001 tablet_replica.cc:330] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:14.000990 10828 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:14.001936 11001 raft_consensus.cc:2229] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:14.002277 10940 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:14.002205 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.003306 11001 raft_consensus.cc:2258] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:14.004477 11001 ts_tablet_manager.cc:1875] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:14.014235 11004 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.020018 7666 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.022593 7535 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.026978 7929 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.027669 7140 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.033947 6999 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.037447 7272 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.038698 9543 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.042508 11001 ts_tablet_manager.cc:1888] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:14.043889 11001 log.cc:1196] T 2dc517698d5a4ae19525c96334ce7417 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:14.045094 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 2dc517698d5a4ae19525c96334ce7417 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:14.046566 7796 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.054780 7401 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:14.057528 7140 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.058621 7929 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.059252 10828 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.059548 9543 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.060576 11002 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.061004 7666 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.061082 11002 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.061666 10998 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.062515 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.063227 7272 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:14.063443 10998 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.063855 11002 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:14.064105 11004 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.066026 7666 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.066779 7796 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.068147 7272 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.071760 9543 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.073431 7000 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.077188 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60cf39e6be52426c93c1534754be7264 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:14.079597 7140 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.081274 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cf39e6be52426c93c1534754be7264 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:14.082113 11001 tablet_replica.cc:330] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:14.082357 11001 raft_consensus.cc:2229] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:14.082995 11001 raft_consensus.cc:2258] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:14.084302 11001 ts_tablet_manager.cc:1875] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:14.086740 7929 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:14.093452 11001 ts_tablet_manager.cc:1888] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:14.093605 11001 log.cc:1196] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/60cf39e6be52426c93c1534754be7264
I20240501 14:03:14.095453 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 60cf39e6be52426c93c1534754be7264 (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:14.142033 11025 ts_tablet_manager.cc:902] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:14.143978 11025 tablet_copy_client.cc:308] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:14.151218 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet 0d2a7bfaac904352a9c9c12c09b563f5 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:60165)
I20240501 14:03:14.151418 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet 0d2a7bfaac904352a9c9c12c09b563f5 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:60165: session id = cc3a47f051a546a5a7788d088f061dd5-0d2a7bfaac904352a9c9c12c09b563f5
I20240501 14:03:14.153954 7818 tablet_copy_source_session.cc:205] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:14.155294 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:14.160737 11025 tablet_copy_client.cc:783] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:14.160993 11025 tablet_copy_client.cc:647] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:14.162881 11025 tablet_copy_client.cc:523] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:14.165868 11025 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:14.197870 11025 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:14.198294 11025 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:14.198561 11025 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20240501 14:03:14.199293 11025 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:14.199570 11025 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:14.199790 11025 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:14.200366 11025 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:14.201210 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-0d2a7bfaac904352a9c9c12c09b563f5 received from {username='slave'} at 127.22.5.200:60165
I20240501 14:03:14.201375 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-0d2a7bfaac904352a9c9c12c09b563f5 on tablet 0d2a7bfaac904352a9c9c12c09b563f5 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:14.331300 7271 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } }
I20240501 14:03:14.336180 7140 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.337806 7401 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.339002 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.340099 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.341925 7929 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.343072 10937 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.360921 10937 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.361678 7401 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.366338 7796 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.366966 11041 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.367369 7535 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.368235 7140 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.369486 7666 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.374747 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.375445 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.377744 6997 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.385252 7929 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.386431 7797 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.399513 7271 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:14.399945 7666 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.403074 7535 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } }
I20240501 14:03:14.411154 7666 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.411885 7796 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.412649 7140 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.413396 11041 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.413813 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.414255 11040 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.419024 11040 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.420218 7929 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.420959 7401 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.420958 10937 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.422415 10937 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.423071 7140 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.425033 7535 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.425984 10937 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.430500 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ab9f466bd0244debff3e228ba99eb51 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:14.432049 7666 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.432483 7796 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
W20240501 14:03:14.435804 7206 consensus_peers.cc:485] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab9f466bd0244debff3e228ba99eb51. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:14.438844 7000 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.449573 7929 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.451865 7401 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.455158 7535 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:14.477401 7929 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:14.480659 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.480931 7401 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.481606 10997 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.481773 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.482692 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.483031 10997 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.482870 7796 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.484338 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.493100 7271 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.493973 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.497541 7535 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:14.498272 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:14.511668 10997 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.514657 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.514822 7797 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.517469 7666 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.518656 7534 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.522790 6993 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.524152 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.554373 7271 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:14.557301 7929 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:14.562441 7534 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:03:14.562516 7865 consensus_peers.cc:485] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5396b2071184070bcb14763e1df7b5f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:14.563023 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.562963 11064 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.564492 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.564703 7401 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.564770 11071 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.565074 7797 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.565491 7271 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:14.565500 11069 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.566439 11069 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.567190 11069 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.568214 11069 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:14.569720 11065 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.573868 6985 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c5396b2071184070bcb14763e1df7b5f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:14.573397 7271 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.575443 7534 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.576335 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.581629 7796 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.582448 6999 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.587946 7666 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.598726 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:14.616091 9543 raft_consensus.cc:1203] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:15.004951 11102 ts_tablet_manager.cc:902] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:15.006134 11102 tablet_copy_client.cc:308] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:15.012076 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 1ab9f466bd0244debff3e228ba99eb51 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:57287)
I20240501 14:03:15.012223 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 1ab9f466bd0244debff3e228ba99eb51 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:57287: session id = 1c61c125cd144590ae1e54a2f87d9187-1ab9f466bd0244debff3e228ba99eb51
I20240501 14:03:15.014709 7293 tablet_copy_source_session.cc:205] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:15.016054 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:15.020534 11102 tablet_copy_client.cc:783] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:15.020792 11102 tablet_copy_client.cc:647] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:15.022543 11102 tablet_copy_client.cc:523] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:15.026732 11102 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:15.060134 10974 raft_consensus.cc:1050] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:15.060549 11102 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:15.060950 11102 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:15.061250 11102 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.008s
I20240501 14:03:15.061272 10974 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:15.061990 11102 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:15.062273 11102 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:15.062508 11102 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:15.063326 11102 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:15.064145 9543 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.064649 7293 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-1ab9f466bd0244debff3e228ba99eb51 received from {username='slave'} at 127.22.5.202:57287
I20240501 14:03:15.064664 7666 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.065405 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.065852 7534 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.065959 7402 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.066638 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.066857 7140 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.064870 7293 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-1ab9f466bd0244debff3e228ba99eb51 on tablet 1ab9f466bd0244debff3e228ba99eb51 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:15.068038 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.068647 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.069129 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.103296 11118 ts_tablet_manager.cc:902] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:15.106256 7926 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.106906 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.106370 11111 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.107872 7272 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:15.110287 7666 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.111002 10962 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.115658 7272 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.116320 7140 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.124155 6993 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.124608 9543 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.125113 7534 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.134982 7401 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.144575 7926 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.146209 11118 tablet_copy_client.cc:308] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:15.147269 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet c5396b2071184070bcb14763e1df7b5f from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:33315)
I20240501 14:03:15.147464 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet c5396b2071184070bcb14763e1df7b5f from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:33315: session id = 41ab226fa6b7403b8b44642e9470517d-c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:15.152529 7949 tablet_copy_source_session.cc:205] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:15.156163 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:15.163128 11118 tablet_copy_client.cc:783] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:15.163475 11118 tablet_copy_client.cc:647] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:15.165446 11118 tablet_copy_client.cc:523] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:15.168712 11118 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:15.206501 11118 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:15.207008 11118 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:15.207348 11118 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.039s user 0.031s sys 0.007s
I20240501 14:03:15.208142 11118 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:15.208456 11118 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:15.208806 11118 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:15.209645 11118 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:03:15.210397 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-c5396b2071184070bcb14763e1df7b5f received from {username='slave'} at 127.22.5.201:33315
I20240501 14:03:15.210562 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-c5396b2071184070bcb14763e1df7b5f on tablet c5396b2071184070bcb14763e1df7b5f with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:15.465159 9677 raft_consensus.cc:1203] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:15.487206 9813 raft_consensus.cc:1203] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:15.651861 7796 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:15.655305 9543 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.655653 7401 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.655750 7140 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.656092 10974 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.656673 7664 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.658007 7272 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.658303 11167 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.658958 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.659695 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.660341 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.663892 7926 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:15.664621 11038 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:15.666050 11038 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.666673 7140 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.668087 7926 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.669580 7272 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.674755 7402 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.677659 6995 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.681845 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d2a7bfaac904352a9c9c12c09b563f5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:15.683571 9543 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.684242 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:15.696260 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:15.704460 11181 tablet_replica.cc:330] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:15.704770 11181 raft_consensus.cc:2229] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:15.705188 11181 raft_consensus.cc:2258] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:15.706143 11181 ts_tablet_manager.cc:1875] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:15.711158 11181 ts_tablet_manager.cc:1888] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:15.711308 11181 log.cc:1196] T 0d2a7bfaac904352a9c9c12c09b563f5 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d2a7bfaac904352a9c9c12c09b563f5
I20240501 14:03:15.712309 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 0d2a7bfaac904352a9c9c12c09b563f5 (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
I20240501 14:03:15.928453 7402 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } }
I20240501 14:03:15.948336 7929 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.948613 7796 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.949167 11174 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:15.950161 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:15.952374 7666 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.956255 11191 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:03:15.957562 7534 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.958230 11174 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:15.961903 7272 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.962579 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:15.965842 7140 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:03:15.966537 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:03:15.973006 7666 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.975586 7929 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.972460 11191 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.978783 6993 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.987264 7796 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.993422 7534 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.995491 7272 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:15.996313 7140 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } }
I20240501 14:03:16.000720 11065 raft_consensus.cc:1050] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:16.002272 11065 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:16.006135 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.007066 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.007277 9677 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.008241 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.010922 7402 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:16.015408 7534 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.015918 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.017901 7929 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.020596 7140 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.020545 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.021688 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.025085 7796 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.025686 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.025372 11174 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.029914 7534 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.030659 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.030237 7139 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.035024 7401 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.035606 6984 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5efbd81b803841a288eed30cb91dad31 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:03:16.037497 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.037727 7929 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.037892 7535 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.043344 6998 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.054693 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.054736 7139 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.054862 7664 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.055279 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.056160 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.056594 7797 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.058324 7271 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:16.058279 11174 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.058858 11174 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.061393 7797 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.070477 11174 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:16.071067 7666 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.072584 7271 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:16.073329 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.074445 11214 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
W20240501 14:03:16.074311 7338 consensus_peers.cc:485] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5efbd81b803841a288eed30cb91dad31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:16.084534 11218 raft_consensus.cc:1050] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:16.090103 11218 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:16.090723 6992 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.091506 7272 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.100284 7796 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.103003 7140 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.103821 11211 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.112890 9677 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.113613 7535 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.123811 7929 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.124608 11219 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.125195 7666 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.126063 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.144176 7926 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:16.151361 9677 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.151542 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.152231 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.152705 11214 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.154454 7401 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.155557 7139 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.157286 11065 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.158408 7796 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.158645 10988 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.158901 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.159080 11214 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.160687 11065 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.161607 11202 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.162989 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.166743 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5396b2071184070bcb14763e1df7b5f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:16.167213 7272 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.167393 7139 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.167572 7796 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.170518 7666 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.170210 9677 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.174326 6999 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:16.183595 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet c5396b2071184070bcb14763e1df7b5f with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:16.184011 11181 tablet_replica.cc:330] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:16.184222 11181 raft_consensus.cc:2229] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:16.184492 11181 raft_consensus.cc:2258] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:16.186517 11181 ts_tablet_manager.cc:1875] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:16.195011 11181 ts_tablet_manager.cc:1888] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:16.197808 7534 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.199065 11181 log.cc:1196] T c5396b2071184070bcb14763e1df7b5f P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:16.199469 11218 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.200925 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet c5396b2071184070bcb14763e1df7b5f (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
W20240501 14:03:16.203310 7075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:33993, user_credentials={real_user=slave}} blocked reactor thread for 48685.8us
I20240501 14:03:16.219137 9813 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.219199 7666 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.219192 7402 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.220062 7797 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:16.221643 11211 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.222820 11211 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.223692 11211 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.224455 11220 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:16.241662 11211 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.242280 7401 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.247248 6993 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.248927 7139 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.250525 7929 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.257542 7534 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.260696 7666 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.261047 7797 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.270341 9813 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.274016 7272 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:16.277019 7929 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.277529 7797 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.277856 7139 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.278101 11161 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.278630 11161 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.279191 11220 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.282325 7401 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.282431 11220 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.283154 7666 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.283207 7796 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.287464 11161 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.287685 11219 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.289369 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab9f466bd0244debff3e228ba99eb51 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:16.290513 7667 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.291726 7401 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.296497 9812 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.298368 11218 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.298662 6993 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.299686 7929 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.300472 9813 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.302206 7140 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:16.311746 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:16.312382 11181 tablet_replica.cc:330] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:16.312743 11181 raft_consensus.cc:2229] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:16.313160 11181 raft_consensus.cc:2258] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:16.316670 11181 ts_tablet_manager.cc:1875] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:16.323648 11181 ts_tablet_manager.cc:1888] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:16.323952 11181 log.cc:1196] T 1ab9f466bd0244debff3e228ba99eb51 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ab9f466bd0244debff3e228ba99eb51
I20240501 14:03:16.324859 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 1ab9f466bd0244debff3e228ba99eb51 (table rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]) successfully deleted
I20240501 14:03:16.604980 11252 ts_tablet_manager.cc:902] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:16.606264 11252 tablet_copy_client.cc:308] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:16.612402 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 5efbd81b803841a288eed30cb91dad31 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:35611)
I20240501 14:03:16.612594 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 5efbd81b803841a288eed30cb91dad31 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:35611: session id = 1c61c125cd144590ae1e54a2f87d9187-5efbd81b803841a288eed30cb91dad31
I20240501 14:03:16.614979 7425 tablet_copy_source_session.cc:205] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:16.616185 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:16.620507 11252 tablet_copy_client.cc:783] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:16.620764 11252 tablet_copy_client.cc:647] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:16.622613 11252 tablet_copy_client.cc:523] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:16.625304 11252 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:16.660149 11252 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:16.660657 11252 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:16.660980 11252 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.008s
I20240501 14:03:16.661810 11252 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:16.662160 11252 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:16.662397 11252 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:16.663082 11252 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:03:16.663946 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-5efbd81b803841a288eed30cb91dad31 received from {username='slave'} at 127.22.5.202:35611
I20240501 14:03:16.664068 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-5efbd81b803841a288eed30cb91dad31 on tablet 5efbd81b803841a288eed30cb91dad31 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:17.181033 9813 raft_consensus.cc:1203] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:03:17.689851 11278 raft_consensus.cc:1050] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:17.690873 11278 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:17.693944 9813 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.693955 7797 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.694444 7667 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.694537 7534 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.694842 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.695436 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.695749 11293 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.696228 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.699223 7929 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.700032 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.700145 7140 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.704661 11086 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.721259 7272 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:17.722295 11294 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:17.723096 11086 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.729099 7667 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.729012 7929 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.730612 7797 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.731735 6998 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:17.739874 7272 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.740584 9813 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.745048 7140 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.748198 7534 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.750298 7402 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:17.755021 7667 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.755800 7929 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.756417 7797 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.756992 7272 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.758474 9812 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.758994 11294 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.762360 11294 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.763680 11294 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.764057 11278 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.765859 11278 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.769308 11295 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.770073 7140 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.770850 11303 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.772073 7926 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.772568 7140 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.773320 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5efbd81b803841a288eed30cb91dad31 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:17.776841 7797 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.776386 6997 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:17.782012 9812 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.782007 7271 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.788230 7667 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:17.795506 7515 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efbd81b803841a288eed30cb91dad31 with delete_type TABLET_DATA_TOMBSTONED (TS 704fb2d37343406790224dc76b40f210 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58278
I20240501 14:03:17.799294 11315 tablet_replica.cc:330] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: stopping tablet replica
I20240501 14:03:17.799579 11315 raft_consensus.cc:2229] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:17.799925 11315 raft_consensus.cc:2258] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:17.801492 11315 ts_tablet_manager.cc:1875] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:17.807783 11315 ts_tablet_manager.cc:1888] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:17.807911 11315 log.cc:1196] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-3/wal/wals/5efbd81b803841a288eed30cb91dad31
I20240501 14:03:17.808748 6985 catalog_manager.cc:4830] TS 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): tablet 5efbd81b803841a288eed30cb91dad31 (table rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]) successfully deleted
I20240501 14:03:18.135250 7120 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.163332 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.179689 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.218073 7251 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.250340 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.276098 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.284595 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:18.329510 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.341043 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.362708 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:18.673549 7120 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.695294 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.726645 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.727293 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.788935 7251 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.789225 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.875255 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.902870 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:18.905822 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:18.939595 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:19.073724 7534 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.078455 7140 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.078656 7929 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.079142 7402 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.079377 11415 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.079833 7271 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.080391 9543 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.080432 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.081645 7797 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.082649 11415 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.083016 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.083526 11415 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.083911 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:19.094980 7469 consensus_peers.cc:485] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f56b1674c3ec4bdc9c2bd82a231d0112. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.097097 11414 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.100374 9543 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.104756 7140 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.105275 7929 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.106509 6992 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:19.111460 7402 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.134974 7271 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.137732 7796 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.154278 7271 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:19.158599 7534 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.159804 7929 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.161129 11220 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.173787 9543 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.174746 11220 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.175324 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.175509 7402 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.175719 7796 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.175875 11220 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.176486 11435 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.177974 11436 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 14:03:19.178728 11220 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.180066 11220 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.180819 7926 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
W20240501 14:03:19.182076 7209 consensus_peers.cc:485] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253a0abd26814fb29f9adede664c5988. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.187527 7534 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.187906 7140 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.194492 6998 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.211448 7796 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.216259 7402 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.218655 9543 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.219810 7926 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:19.231222 7796 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.231290 9813 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:19.232348 7862 consensus_peers.cc:485] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab7d2ddba3440f0a1311f9eda7d3930. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.232504 7402 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.232537 7667 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.232793 11214 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.233101 7271 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.233088 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.233711 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.233870 7140 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.234333 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.234606 11455 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.236207 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.248160 11202 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.254261 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.260905 7796 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.262229 7667 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.264668 9813 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.265847 6991 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:19.269506 7402 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.286053 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.302641 7926 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.305693 9677 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.305761 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:19.306223 7862 consensus_peers.cc:485] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5396b2071184070bcb14763e1df7b5f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.306520 11455 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.307366 11462 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.307855 7796 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.308851 11462 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.309196 7402 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.309273 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.310240 11454 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.310411 7271 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.310957 11454 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.313658 11202 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.318313 11454 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.323858 7271 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.327292 9678 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.331171 6999 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.333719 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.334241 7666 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.340134 7796 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.340463 7402 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.363863 7402 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:19.367050 7272 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.367059 7140 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.367616 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:19.368065 7341 consensus_peers.cc:485] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cf39e6be52426c93c1534754be7264. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.368175 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.370154 7926 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.371832 9543 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.372053 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.372956 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.373740 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.377939 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.394789 11293 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.400192 7666 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.400868 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.400360 9544 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.403038 7666 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.406857 6999 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.414750 7139 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.419634 7926 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.430996 7272 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.441638 7796 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.471478 7666 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:19.475067 7139 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.475155 9813 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.476047 7402 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.476184 7272 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.476794 11075 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.477454 11075 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.477687 7796 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.477784 11476 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.478343 11476 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.481078 7926 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.481848 11075 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 14:03:19.493685 11476 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.501895 11511 ts_tablet_manager.cc:902] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547)
I20240501 14:03:19.504667 11476 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:19.506505 7600 consensus_peers.cc:485] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e104d44c5fa4a29be94a33cf7e93ba6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.508015 9813 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.511462 7926 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.511611 7139 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.513592 7796 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.519783 6995 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.521029 7272 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.528179 7402 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.537009 11511 tablet_copy_client.cc:308] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.196:37547
I20240501 14:03:19.553092 7556 tablet_copy_service.cc:140] P 704fb2d37343406790224dc76b40f210: Received BeginTabletCopySession request for tablet f56b1674c3ec4bdc9c2bd82a231d0112 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:54883)
I20240501 14:03:19.553329 7556 tablet_copy_service.cc:161] P 704fb2d37343406790224dc76b40f210: Beginning new tablet copy session on tablet f56b1674c3ec4bdc9c2bd82a231d0112 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:54883: session id = 1c61c125cd144590ae1e54a2f87d9187-f56b1674c3ec4bdc9c2bd82a231d0112
I20240501 14:03:19.555653 7556 tablet_copy_source_session.cc:205] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:19.556978 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56b1674c3ec4bdc9c2bd82a231d0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:19.561856 11511 tablet_copy_client.cc:783] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:19.562086 11511 tablet_copy_client.cc:647] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:19.562217 7139 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.565299 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:19.565389 7074 consensus_peers.cc:485] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2264386e494644aa84e8109c816aaefa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.566136 11511 tablet_copy_client.cc:523] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:19.566213 7534 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.566279 11446 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.567332 7402 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.567831 11446 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.568037 7796 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.568554 11363 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.572115 7926 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.572755 11363 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.573292 11363 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.575747 11511 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:19.591442 11508 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.593627 11532 ts_tablet_manager.cc:902] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:19.603468 7272 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.603991 11446 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.603264 7401 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.602813 7926 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.610549 7535 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.610863 6995 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.614609 7796 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.615300 9678 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.630827 7272 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.647871 11532 tablet_copy_client.cc:308] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:19.668099 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 253a0abd26814fb29f9adede664c5988 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:38919)
I20240501 14:03:19.668313 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 253a0abd26814fb29f9adede664c5988 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:38919: session id = 41ab226fa6b7403b8b44642e9470517d-253a0abd26814fb29f9adede664c5988
I20240501 14:03:19.671380 7293 tablet_copy_source_session.cc:205] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:19.673491 11532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:19.680728 11532 tablet_copy_client.cc:783] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:19.681083 11532 tablet_copy_client.cc:647] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:19.681421 7926 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:19.686257 11532 tablet_copy_client.cc:523] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:19.687032 9813 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:19.687264 7865 consensus_peers.cc:485] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd9f7d11351b4f8d84021720488abf25. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.687795 7401 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.688212 11202 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.689677 11532 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:19.689690 11214 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.691260 11511 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:19.691735 11511 tablet_bootstrap.cc:492] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:19.692014 11511 ts_tablet_manager.cc:1367] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.123s user 0.047s sys 0.003s
I20240501 14:03:19.692890 7140 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.692901 11511 raft_consensus.cc:348] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.693322 11511 raft_consensus.cc:729] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:19.693379 7272 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.693528 11511 consensus_queue.cc:260] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.693770 11454 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.694294 11454 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.694445 11511 ts_tablet_manager.cc:1398] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:19.695384 7556 tablet_copy_service.cc:342] P 704fb2d37343406790224dc76b40f210: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-f56b1674c3ec4bdc9c2bd82a231d0112 received from {username='slave'} at 127.22.5.202:54883
I20240501 14:03:19.695510 7556 tablet_copy_service.cc:433] P 704fb2d37343406790224dc76b40f210: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-f56b1674c3ec4bdc9c2bd82a231d0112 on tablet f56b1674c3ec4bdc9c2bd82a231d0112 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:19.698757 7534 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.700165 7796 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.700978 11454 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.701493 11454 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.709005 11454 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.711783 7272 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.713402 7402 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.717578 9813 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.718106 6991 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.719871 7140 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.733016 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.736408 7796 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:19.754705 7926 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:19.758129 11532 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:19.758209 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.758648 11532 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:19.758811 7272 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.758836 7402 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.758960 11532 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.069s user 0.047s sys 0.006s
I20240501 14:03:19.759945 11214 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.759829 11532 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:19.760185 9678 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.760257 11532 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:19.760526 11455 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.760551 11532 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:19.761235 11532 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 14:03:19.761139 11455 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:19.761453 7862 consensus_peers.cc:485] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9812475b87e9492d8c9450b77d34ea6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.761713 11455 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.763110 7293 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-253a0abd26814fb29f9adede664c5988 received from {username='slave'} at 127.22.5.201:38919
I20240501 14:03:19.763243 7293 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-253a0abd26814fb29f9adede664c5988 on tablet 253a0abd26814fb29f9adede664c5988 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:19.773913 7796 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.774494 11214 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.777633 11455 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.780810 9677 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.780970 7797 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.786669 6991 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.792083 7401 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.797883 7666 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.797890 7140 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.801905 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.807135 11563 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.808812 11511 ts_tablet_manager.cc:902] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:19.821157 7666 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:19.829080 7140 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:19.829821 11576 ts_tablet_manager.cc:902] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:19.834335 9678 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.834721 7401 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.835397 11535 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.836058 11535 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.843438 7272 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.843988 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.850705 7926 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.851320 7797 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.851927 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.852098 7666 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:19.852344 11535 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:19.852815 7074 consensus_peers.cc:485] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b227b60efb7343bba8c4e89f81c3f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.852815 11535 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240501 14:03:19.857061 11532 ts_tablet_manager.cc:902] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:19.890349 11511 tablet_copy_client.cc:308] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:19.891188 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet c5396b2071184070bcb14763e1df7b5f from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:60515)
I20240501 14:03:19.891387 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet c5396b2071184070bcb14763e1df7b5f from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:60515: session id = 1c61c125cd144590ae1e54a2f87d9187-c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:19.892700 11508 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.894270 7949 tablet_copy_source_session.cc:205] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:19.899260 7667 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.899581 7797 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.904650 6991 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:19.908650 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:19.917969 7926 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.919513 11511 tablet_copy_client.cc:783] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:19.919802 11511 tablet_copy_client.cc:647] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:19.922086 11511 tablet_copy_client.cc:523] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:19.926463 11511 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:19.950191 11591 ts_tablet_manager.cc:902] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:19.954916 9813 raft_consensus.cc:1203] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 14:03:19.965263 11585 log.cc:924] Time spent T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20240501 14:03:19.967298 7272 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.978830 7402 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.978725 9678 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:19.989717 11576 tablet_copy_client.cc:308] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:19.992348 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:57433)
I20240501 14:03:19.992514 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:57433: session id = cc3a47f051a546a5a7788d088f061dd5-1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:19.995577 7949 tablet_copy_source_session.cc:205] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:19.997375 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.013797 11576 tablet_copy_client.cc:783] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.014156 11576 tablet_copy_client.cc:647] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.018147 9678 raft_consensus.cc:1203] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.019111 11532 tablet_copy_client.cc:308] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:20.020013 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 60cf39e6be52426c93c1534754be7264 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:46003)
I20240501 14:03:20.020198 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 60cf39e6be52426c93c1534754be7264 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:46003: session id = 41ab226fa6b7403b8b44642e9470517d-60cf39e6be52426c93c1534754be7264
I20240501 14:03:20.023308 7425 tablet_copy_source_session.cc:205] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.025350 11591 tablet_copy_client.cc:308] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:42421
I20240501 14:03:20.025801 11576 tablet_copy_client.cc:523] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.027307 11532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.031152 11576 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:20.048566 11532 tablet_copy_client.cc:783] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.048835 11532 tablet_copy_client.cc:647] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.050664 11532 tablet_copy_client.cc:523] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.054276 11532 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:20.064278 11602 ts_tablet_manager.cc:902] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:20.065548 11511 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.068286 11511 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:20.068590 11511 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.142s user 0.052s sys 0.003s
I20240501 14:03:20.069449 11511 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.069777 11511 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:20.070045 11511 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.073613 11511 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:03:20.076881 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-c5396b2071184070bcb14763e1df7b5f received from {username='slave'} at 127.22.5.202:60515
I20240501 14:03:20.077051 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-c5396b2071184070bcb14763e1df7b5f on tablet c5396b2071184070bcb14763e1df7b5f with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:20.085237 7687 tablet_copy_service.cc:140] P db6b830d9bbe4451bd349c6ad6e14bab: Received BeginTabletCopySession request for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:34765)
I20240501 14:03:20.085430 7687 tablet_copy_service.cc:161] P db6b830d9bbe4451bd349c6ad6e14bab: Beginning new tablet copy session on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:34765: session id = cc3a47f051a546a5a7788d088f061dd5-4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:20.088429 7687 tablet_copy_source_session.cc:205] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.090869 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.099670 11591 tablet_copy_client.cc:783] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.100229 11591 tablet_copy_client.cc:647] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.102427 11591 tablet_copy_client.cc:523] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.127885 11576 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.128382 11576 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:20.128710 11576 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.098s user 0.055s sys 0.010s
I20240501 14:03:20.131289 11576 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.131783 11576 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:20.132103 11576 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.134836 11591 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:20.150897 11576 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.022s user 0.007s sys 0.011s
I20240501 14:03:20.158133 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-1ab7d2ddba3440f0a1311f9eda7d3930 received from {username='slave'} at 127.22.5.200:57433
I20240501 14:03:20.158322 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-1ab7d2ddba3440f0a1311f9eda7d3930 on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:20.163317 11532 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.163897 11532 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:20.164189 11532 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.110s user 0.067s sys 0.012s
I20240501 14:03:20.165074 11602 tablet_copy_client.cc:308] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:20.165486 11532 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:20.165715 11532 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:20.165912 11532 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:20.166143 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet 2264386e494644aa84e8109c816aaefa from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:48843)
I20240501 14:03:20.166307 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet 2264386e494644aa84e8109c816aaefa from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:48843: session id = 1c61c125cd144590ae1e54a2f87d9187-2264386e494644aa84e8109c816aaefa
I20240501 14:03:20.166733 11532 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:20.169057 7161 tablet_copy_source_session.cc:205] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.172103 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-60cf39e6be52426c93c1534754be7264 received from {username='slave'} at 127.22.5.201:46003
I20240501 14:03:20.172312 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-60cf39e6be52426c93c1534754be7264 on tablet 60cf39e6be52426c93c1534754be7264 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:20.174296 11532 ts_tablet_manager.cc:902] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:20.187557 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.200928 11532 tablet_copy_client.cc:308] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:20.202147 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet bd9f7d11351b4f8d84021720488abf25 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:33315)
I20240501 14:03:20.202348 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet bd9f7d11351b4f8d84021720488abf25 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:33315: session id = 41ab226fa6b7403b8b44642e9470517d-bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:20.204818 7949 tablet_copy_source_session.cc:205] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.207319 11576 ts_tablet_manager.cc:902] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:20.212517 11532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.216275 11602 tablet_copy_client.cc:783] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.216552 11602 tablet_copy_client.cc:647] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.218605 9813 raft_consensus.cc:1203] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.221890 11602 tablet_copy_client.cc:523] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.222836 11532 tablet_copy_client.cc:783] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.223124 11532 tablet_copy_client.cc:647] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.229992 11602 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:20.233676 11532 tablet_copy_client.cc:523] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.237921 11532 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:20.257557 11576 tablet_copy_client.cc:308] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:20.258548 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 9812475b87e9492d8c9450b77d34ea6b from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:57433)
I20240501 14:03:20.258751 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 9812475b87e9492d8c9450b77d34ea6b from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:57433: session id = cc3a47f051a546a5a7788d088f061dd5-9812475b87e9492d8c9450b77d34ea6b
I20240501 14:03:20.274163 9678 raft_consensus.cc:1203] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.279021 11591 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.282755 11591 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:20.283139 11591 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.148s user 0.080s sys 0.042s
I20240501 14:03:20.283973 11591 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.284515 11591 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:20.285439 11591 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.287535 11591 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:03:20.289820 7687 tablet_copy_service.cc:342] P db6b830d9bbe4451bd349c6ad6e14bab: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-4e104d44c5fa4a29be94a33cf7e93ba6 received from {username='slave'} at 127.22.5.200:34765
I20240501 14:03:20.290014 7687 tablet_copy_service.cc:433] P db6b830d9bbe4451bd349c6ad6e14bab: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-4e104d44c5fa4a29be94a33cf7e93ba6 on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:20.298905 7949 tablet_copy_source_session.cc:205] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.300573 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9812475b87e9492d8c9450b77d34ea6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.307163 11576 tablet_copy_client.cc:783] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.307443 11576 tablet_copy_client.cc:647] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.310302 11576 tablet_copy_client.cc:523] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.313897 11576 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:20.332167 11602 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.332666 11602 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:20.332989 11602 ts_tablet_manager.cc:1367] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.103s user 0.056s sys 0.001s
I20240501 14:03:20.333863 11602 raft_consensus.cc:348] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.334226 11602 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:20.334444 11602 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.335119 11602 ts_tablet_manager.cc:1398] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:20.336771 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-2264386e494644aa84e8109c816aaefa received from {username='slave'} at 127.22.5.202:48843
I20240501 14:03:20.336917 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-2264386e494644aa84e8109c816aaefa on tablet 2264386e494644aa84e8109c816aaefa with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:20.347467 11511 ts_tablet_manager.cc:902] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:20.368018 11532 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.368522 11532 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:20.368811 11532 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.131s user 0.063s sys 0.040s
I20240501 14:03:20.369702 11532 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:20.370049 11532 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:20.370255 11532 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:20.371687 11532 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:20.373817 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-bd9f7d11351b4f8d84021720488abf25 received from {username='slave'} at 127.22.5.201:33315
I20240501 14:03:20.374035 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-bd9f7d11351b4f8d84021720488abf25 on tablet bd9f7d11351b4f8d84021720488abf25 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:20.382944 9543 raft_consensus.cc:1203] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.391001 11576 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.391702 11576 tablet_bootstrap.cc:492] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:20.392069 11576 ts_tablet_manager.cc:1367] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.078s user 0.055s sys 0.009s
I20240501 14:03:20.392998 11576 raft_consensus.cc:348] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.393687 11576 raft_consensus.cc:729] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:20.394057 11576 consensus_queue.cc:260] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:20.396159 11576 ts_tablet_manager.cc:1398] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:03:20.397578 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-9812475b87e9492d8c9450b77d34ea6b received from {username='slave'} at 127.22.5.200:57433
I20240501 14:03:20.397780 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-9812475b87e9492d8c9450b77d34ea6b on tablet 9812475b87e9492d8c9450b77d34ea6b with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:20.408113 11511 tablet_copy_client.cc:308] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:20.409112 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:48843)
I20240501 14:03:20.409370 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:48843: session id = 1c61c125cd144590ae1e54a2f87d9187-b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:20.412360 7161 tablet_copy_source_session.cc:205] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.413800 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.420605 11511 tablet_copy_client.cc:783] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.420960 11511 tablet_copy_client.cc:647] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.424983 11414 raft_consensus.cc:1050] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:20.426610 11414 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:20.429450 11511 tablet_copy_client.cc:523] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.435760 11511 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:20.438568 9543 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.438839 7796 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.437269 9813 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.440590 11594 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.441265 9543 raft_consensus.cc:1203] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.441290 11594 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.441799 11594 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.450491 7929 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.451771 7140 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.457484 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.458405 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.460533 7402 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.465730 7272 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.465970 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.467275 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.477413 9812 raft_consensus.cc:1203] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.479686 11220 raft_consensus.cc:1050] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:20.481109 11220 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:20.559125 11511 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.559634 11511 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:20.559969 11511 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.124s user 0.075s sys 0.001s
I20240501 14:03:20.560845 11511 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.561170 11511 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:20.561396 11511 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:20.562160 11511 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 14:03:20.563400 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-b227b60efb7343bba8c4e89f81c3f6ff received from {username='slave'} at 127.22.5.202:48843
I20240501 14:03:20.563555 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-b227b60efb7343bba8c4e89f81c3f6ff on tablet b227b60efb7343bba8c4e89f81c3f6ff with peer 1c61c125cd144590ae1e54a2f87d9187
W20240501 14:03:20.592747 7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:41451, user_credentials={real_user=slave}} blocked reactor thread for 139410us
I20240501 14:03:20.594231 11598 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.598204 7926 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.598258 7534 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.599007 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.598254 7929 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.599697 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.601481 9544 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.604418 9813 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.604710 7272 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.608776 9678 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.610208 11445 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.611474 7140 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.613269 6997 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.619935 7796 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.625619 7402 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.644910 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.645954 7797 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.646420 11445 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.649251 9678 raft_consensus.cc:1203] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.650368 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.657148 9544 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.657111 11220 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.657873 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.657804 7797 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.659211 7402 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.659835 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.662883 9678 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.663427 7926 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.666734 7534 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.679801 6996 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.684434 7140 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.712881 9543 raft_consensus.cc:1203] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.711830 7534 consensus_queue.cc:237] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:20.713856 7402 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.717481 9813 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.720049 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.719610 9544 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.729590 9544 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.730099 7796 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.730337 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.730886 11659 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.732343 7926 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.732708 7272 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.732931 7401 raft_consensus.cc:1261] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 704fb2d37343406790224dc76b40f210: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.733528 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.733896 11414 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.735666 11598 consensus_queue.cc:1035] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.736277 11659 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 704fb2d37343406790224dc76b40f210 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.737407 7401 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.737882 7272 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.743924 9813 raft_consensus.cc:1203] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:20.748900 7000 catalog_manager.cc:5484] T f56b1674c3ec4bdc9c2bd82a231d0112 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "704fb2d37343406790224dc76b40f210" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.751209 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f56b1674c3ec4bdc9c2bd82a231d0112 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:20.757299 9813 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.759459 9544 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.757009 7926 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.766402 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet f56b1674c3ec4bdc9c2bd82a231d0112 with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:20.767401 11707 tablet_replica.cc:330] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:20.768858 11707 raft_consensus.cc:2229] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:20.769309 11707 raft_consensus.cc:2258] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:20.769668 7796 raft_consensus.cc:2932] T f56b1674c3ec4bdc9c2bd82a231d0112 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:20.773591 11707 ts_tablet_manager.cc:1875] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:20.782843 11707 ts_tablet_manager.cc:1888] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:20.783046 11707 log.cc:1196] T f56b1674c3ec4bdc9c2bd82a231d0112 P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/f56b1674c3ec4bdc9c2bd82a231d0112
I20240501 14:03:20.784319 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet f56b1674c3ec4bdc9c2bd82a231d0112 (table rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]) successfully deleted
I20240501 14:03:20.803301 11660 raft_consensus.cc:1050] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:20.804539 11660 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:20.807981 7140 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.808228 7272 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.808861 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.809203 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.809361 9678 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.810041 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.811726 7667 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.811987 9544 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.812299 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.812443 7926 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.814258 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.816928 11515 raft_consensus.cc:1050] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:20.817829 7796 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.818445 11515 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:20.819974 11660 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.821141 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.835207 11202 raft_consensus.cc:1050] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:20.843343 7272 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.842922 11716 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.843886 7401 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.843974 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.844456 9543 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.845002 7797 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.844857 9677 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.845863 9813 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.847931 7929 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.848750 7271 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.852380 7926 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.852413 7795 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.853020 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.855927 7667 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.856709 6992 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.866927 7140 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.867579 11455 raft_consensus.cc:1050] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:20.868950 11455 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:20.872095 7139 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.872763 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.875355 9543 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.877964 11655 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.878664 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.836951 11202 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:20.879349 11654 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.883183 11654 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.890367 9678 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.891160 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.894321 9812 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.895110 7667 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.895205 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.895390 7795 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.896582 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.897218 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.914040 7402 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:20.919160 9677 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.919364 7272 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.920727 11715 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.921294 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.921602 7798 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.922003 9544 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.922734 11293 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.923316 7667 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:20.923805 11715 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.925765 11295 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:20.929930 7795 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.939090 6997 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.943568 9544 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.944715 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cf39e6be52426c93c1534754be7264 with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:20.945161 11707 tablet_replica.cc:330] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:20.945359 11707 raft_consensus.cc:2229] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:20.944631 7667 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.945608 11707 raft_consensus.cc:2258] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:20.949699 11707 ts_tablet_manager.cc:1875] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:20.958844 11716 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:20.972692 11707 ts_tablet_manager.cc:1888] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:20.977445 11707 log.cc:1196] T 60cf39e6be52426c93c1534754be7264 P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/60cf39e6be52426c93c1534754be7264
I20240501 14:03:20.978535 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 60cf39e6be52426c93c1534754be7264 (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:20.982687 11654 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:20.987434 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:20.987540 7139 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:20.989104 11742 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:20.988751 7272 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:20.991986 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:20.994518 9543 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:20.998709 7401 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.000793 7000 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:21.018776 9812 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.022678 7796 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
W20240501 14:03:21.018149 7862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.197:42421, user_credentials={real_user=slave}} blocked reactor thread for 121780us
I20240501 14:03:21.029356 7139 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.029939 11563 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.029985 7667 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:21.030254 7402 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.031008 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.031751 9544 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.034142 7796 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.035199 9813 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.035517 7666 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.035970 7402 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.036787 9813 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.036978 7272 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.037555 7271 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.037801 7139 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.038048 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.038390 11728 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.038728 11655 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.039232 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:03:21.041591 7865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.201:33735, user_credentials={real_user=slave}} blocked reactor thread for 61045.1us
I20240501 14:03:21.044078 7926 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.044308 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60cf39e6be52426c93c1534754be7264 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.046126 7272 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.046675 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.046517 9678 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.049077 11515 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.052531 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.052435 7926 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.054374 7929 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.056951 11563 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.060657 7796 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.061328 11654 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.062351 11508 raft_consensus.cc:1050] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:21.063606 11508 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:21.063652 7795 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.069197 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.070659 7139 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.072026 7271 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.073086 6998 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.079813 9813 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.083112 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:21.083041 11726 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.093331 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.099563 7926 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.106144 11563 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.106992 9544 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.110210 11515 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.112476 9542 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.142879 11750 tablet_replica.cc:330] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:21.143489 11750 raft_consensus.cc:2229] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.143908 11750 raft_consensus.cc:2258] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.143975 11563 raft_consensus.cc:1050] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:21.145673 11563 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:21.146082 11750 ts_tablet_manager.cc:1875] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.152721 11740 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.154124 11740 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.157459 9813 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.158921 11526 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.163441 11740 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.166893 7139 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.168037 11741 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.174201 9678 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.176270 11740 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.178397 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.178771 7534 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.179384 11535 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.183315 11535 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.183554 11750 ts_tablet_manager.cc:1888] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.183720 11750 log.cc:1196] T 4e104d44c5fa4a29be94a33cf7e93ba6 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:21.184271 11535 raft_consensus.cc:1050] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:21.185441 11535 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:21.189508 7535 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.189514 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:21.190253 11740 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.196084 11563 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.197950 7795 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.198541 11526 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.201512 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.200812 11455 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.210136 7273 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:21.210891 7401 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.211760 11526 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.213727 9678 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.213723 7000 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.214418 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.215981 9544 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.216348 7926 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.217159 7534 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.217258 11678 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.218379 11401 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.218492 7798 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.219702 11678 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.221411 11683 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.224777 11683 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.231511 7271 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.233006 11508 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 14:03:21.233425 7534 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.244829 9678 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.245628 7402 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.246366 7926 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.247679 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 253a0abd26814fb29f9adede664c5988 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.250649 11777 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.255980 7000 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.259302 7795 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.266218 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 253a0abd26814fb29f9adede664c5988 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:21.266736 11750 tablet_replica.cc:330] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:21.266995 11750 raft_consensus.cc:2229] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.267318 11750 raft_consensus.cc:2258] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.266763 11778 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 14:03:21.265111 11780 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.268877 11750 ts_tablet_manager.cc:1875] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.275771 9544 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.277019 11750 ts_tablet_manager.cc:1888] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.277240 11750 log.cc:1196] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/253a0abd26814fb29f9adede664c5988
I20240501 14:03:21.282138 9812 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.286109 7273 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.287186 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet 253a0abd26814fb29f9adede664c5988 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:21.288436 6997 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.288985 7796 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.289930 9812 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.290884 7796 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.278307 9678 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.298041 11778 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.299631 11778 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.300510 11778 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:03:21.318910 7077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.201:33735, user_credentials={real_user=slave}} blocked reactor thread for 73519.2us
I20240501 14:03:21.319936 9677 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
W20240501 14:03:21.352597 7862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:45423, user_credentials={real_user=slave}} blocked reactor thread for 139574us
I20240501 14:03:21.354012 7401 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.354499 7402 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.357564 9812 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.356992 7273 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.357079 9677 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.358637 7795 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.359391 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.360913 7272 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.380004 7271 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:03:21.382230 7075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:33993, user_credentials={real_user=slave}} blocked reactor thread for 156750us
I20240501 14:03:21.382027 7000 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.383414 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.388270 7929 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.389524 7534 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.390489 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.392216 11773 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.394891 11526 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:03:21.399015 7074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.202:37399, user_credentials={real_user=slave}} blocked reactor thread for 99541.2us
I20240501 14:03:21.399760 7929 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:21.409155 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.409533 9678 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.409791 11202 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.415063 7402 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.417418 11202 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.424115 11781 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.428212 11775 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.430514 7929 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.430807 7271 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.432917 7140 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.435487 9678 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.438441 7664 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.437927 7795 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.442328 9813 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.446383 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.452102 7000 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.452431 9544 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.452476 7929 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.458333 9812 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.463184 7402 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.464453 7401 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.469449 7272 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.483310 11202 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.486627 11739 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.485828 7664 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.489048 7796 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.489872 9812 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.490919 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.491341 11202 raft_consensus.cc:1050] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:21.492597 7667 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.493820 7795 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.499886 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.500845 9812 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.502804 7797 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.503338 11202 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:21.506960 11740 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:03:21.517587 11455 consensus_queue.cc:1159] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b3b866abc97f437e8f24eccb653b3dcc" responder_term: 1 status { last_received { term: 1 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 9 } }
I20240501 14:03:21.518070 7929 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:21.518292 11739 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.518559 11563 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.518957 11739 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.520888 9678 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.521564 7140 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.524019 11739 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.523684 7666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.524981 11739 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.529894 11202 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.531186 11202 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.532784 9544 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.534642 9544 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.534695 7797 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.534826 7273 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.535192 9812 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.535389 7271 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.536974 7405 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.538036 7401 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.538818 7667 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.539521 7797 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:21.548533 11740 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.549491 11202 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.551731 11202 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.552775 11202 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.554378 7665 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.554543 7271 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.563449 11202 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.563328 6998 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.564690 11739 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.566431 11739 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.567514 11202 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.568382 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet c5396b2071184070bcb14763e1df7b5f with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:21.569437 11750 tablet_replica.cc:330] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:21.569691 11750 raft_consensus.cc:2229] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.569983 11750 raft_consensus.cc:2258] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.570191 11739 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.571151 11739 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.573179 11739 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.574193 11739 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:21.577302 9811 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.578665 11750 ts_tablet_manager.cc:1875] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.582031 9812 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.592545 6998 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.600806 11736 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.604266 9543 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.613022 9677 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.614234 7139 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.616518 11750 ts_tablet_manager.cc:1888] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.616667 11750 log.cc:1196] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:21.617722 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5396b2071184070bcb14763e1df7b5f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.621727 6993 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.627339 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:21.629419 7139 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.655579 7664 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.667832 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.671564 11807 tablet_replica.cc:330] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:21.672245 11563 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.673930 11807 raft_consensus.cc:2229] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.675421 7405 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.679368 9812 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.679670 7664 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.681064 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.686092 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.688099 11807 raft_consensus.cc:2258] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.691644 11807 ts_tablet_manager.cc:1875] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.691507 9677 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.705971 11807 ts_tablet_manager.cc:1888] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.706167 11807 log.cc:1196] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:21.715418 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:21.711669 7798 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.722549 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet c5396b2071184070bcb14763e1df7b5f (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
I20240501 14:03:21.753291 11739 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.754027 7140 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.759657 9678 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.761727 6993 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.769382 9544 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.774592 7271 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.771132 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.777146 7401 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.776064 7402 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.784952 7798 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.777020 7667 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.803272 7795 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.814458 7797 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.815205 7929 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:21.819988 9678 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.820850 11739 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.825812 7271 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.825829 9812 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.826866 7534 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.827908 11563 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.828121 7797 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.828495 11563 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.830336 11742 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.831698 7140 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.833310 7926 consensus_queue.cc:237] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:21.833561 11455 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.834996 11739 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.840307 7667 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.840915 9544 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.841328 7402 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.841540 7795 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.841252 11741 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.841969 7271 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.846885 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.847707 9678 raft_consensus.cc:1261] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.848392 11563 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.850085 11739 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.850399 11736 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.851213 11202 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.850950 9812 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.851462 11742 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.852026 11736 consensus_queue.cc:1035] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.852607 7271 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.855657 6993 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.860044 7795 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.863683 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f7d11351b4f8d84021720488abf25 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:21.864303 11750 tablet_replica.cc:330] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:21.864605 11750 raft_consensus.cc:2229] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.866302 11750 raft_consensus.cc:2258] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.867990 11750 ts_tablet_manager.cc:1875] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.868580 11739 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.872714 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd9f7d11351b4f8d84021720488abf25 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.872561 9544 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.874008 9677 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.877833 9678 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.880605 7272 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.881664 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9812475b87e9492d8c9450b77d34ea6b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.881425 7140 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:21.883400 7402 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.888744 7139 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:21.889246 7000 catalog_manager.cc:5484] T 9812475b87e9492d8c9450b77d34ea6b P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.889082 7797 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.894588 7926 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.895911 11781 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.896189 7667 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.896229 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812475b87e9492d8c9450b77d34ea6b with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:21.896670 11807 tablet_replica.cc:330] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:21.896831 11750 ts_tablet_manager.cc:1888] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.896879 11807 raft_consensus.cc:2229] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.896970 11750 log.cc:1196] T bd9f7d11351b4f8d84021720488abf25 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:21.897145 11807 raft_consensus.cc:2258] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.897408 7272 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.898448 11779 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.898959 11784 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.899848 11807 ts_tablet_manager.cc:1875] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.900180 9812 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.907181 7667 raft_consensus.cc:2932] T 9812475b87e9492d8c9450b77d34ea6b P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:21.911378 11779 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.913002 11535 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.914330 7664 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.917246 6986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b227b60efb7343bba8c4e89f81c3f6ff with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:21.918463 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet bd9f7d11351b4f8d84021720488abf25 (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:21.919450 7926 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.920253 6992 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:21.926255 9677 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.926738 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:21.926975 11807 ts_tablet_manager.cc:1888] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.927145 11807 log.cc:1196] T 9812475b87e9492d8c9450b77d34ea6b P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/9812475b87e9492d8c9450b77d34ea6b
I20240501 14:03:21.927632 11750 tablet_replica.cc:330] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:21.927704 7797 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:21.927855 11750 raft_consensus.cc:2229] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.928150 11750 raft_consensus.cc:2258] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.928264 9812 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.929734 11750 ts_tablet_manager.cc:1875] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:21.932349 11781 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.932971 11781 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:21.933920 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 9812475b87e9492d8c9450b77d34ea6b (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:21.937463 7797 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.938329 11750 ts_tablet_manager.cc:1888] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:21.938525 11750 log.cc:1196] T b227b60efb7343bba8c4e89f81c3f6ff P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:21.939846 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet b227b60efb7343bba8c4e89f81c3f6ff (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:21.940295 9677 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:21.925014 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:22.001659 7140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2264386e494644aa84e8109c816aaefa"
dest_uuid: "0d05240051574799acbbe13e21b262b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45170
I20240501 14:03:22.001915 7140 raft_consensus.cc:595] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:22.025077 11535 raft_consensus.cc:979] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: : Instructing follower b3b866abc97f437e8f24eccb653b3dcc to start an election
I20240501 14:03:22.025272 11535 raft_consensus.cc:1067] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Signalling peer b3b866abc97f437e8f24eccb653b3dcc to start an election
I20240501 14:03:22.026314 7271 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2264386e494644aa84e8109c816aaefa"
dest_uuid: "b3b866abc97f437e8f24eccb653b3dcc"
from {username='slave'} at 127.22.5.193:37451
I20240501 14:03:22.026615 7271 raft_consensus.cc:482] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:22.026731 7271 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.029716 7271 raft_consensus.cc:504] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:22.031627 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "0d05240051574799acbbe13e21b262b4"
I20240501 14:03:22.031832 7140 raft_consensus.cc:3032] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:22.031936 7140 raft_consensus.cc:729] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Running, Role: LEADER
I20240501 14:03:22.031911 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "704fb2d37343406790224dc76b40f210"
I20240501 14:03:22.032202 7534 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.032202 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8"
I20240501 14:03:22.032404 7402 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.032301 7140 consensus_queue.cc:260] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:22.033011 7140 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.035138 7402 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.035830 7140 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.037220 7534 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.037443 7797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c8f737f3f6994db297b47fb23dad1b13"
I20240501 14:03:22.037681 7797 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.039515 7797 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.040275 7206 leader_election.cc:302] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 8 voters: 5 yes votes; 0 no votes. yes voters: 0d05240051574799acbbe13e21b262b4, 704fb2d37343406790224dc76b40f210, b3b866abc97f437e8f24eccb653b3dcc, c8f737f3f6994db297b47fb23dad1b13, ceb055f6a1a540dcb61e1e11a157cdb8; no voters:
I20240501 14:03:22.040593 11773 raft_consensus.cc:2781] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:22.040740 11773 raft_consensus.cc:686] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 2 LEADER]: Becoming Leader. State: Replica: b3b866abc97f437e8f24eccb653b3dcc, State: Running, Role: LEADER
I20240501 14:03:22.041056 11773 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:22.042315 7271 leader_election.cc:288] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [CANDIDATE]: Term 2 election: Requested vote from peers 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299), ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999), 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547), c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423), cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993), 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735), 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399)
I20240501 14:03:22.043885 9677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "41ab226fa6b7403b8b44642e9470517d"
I20240501 14:03:22.043916 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "cc7e7386a9e540eabc4648e147073f1d"
I20240501 14:03:22.044152 7926 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.044226 9677 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.046427 7926 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.047158 9677 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.047595 9812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2264386e494644aa84e8109c816aaefa" candidate_uuid: "b3b866abc97f437e8f24eccb653b3dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1c61c125cd144590ae1e54a2f87d9187"
I20240501 14:03:22.047897 9812 raft_consensus.cc:3037] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:22.050113 9812 raft_consensus.cc:2454] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b866abc97f437e8f24eccb653b3dcc in term 2.
I20240501 14:03:22.053265 6997 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: term changed from 1 to 2, leader changed from 0d05240051574799acbbe13e21b262b4 (127.22.5.193) to b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194). New cstate: current_term: 2 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:22.177047 7797 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:22.180634 9544 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.181198 7140 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.181483 7402 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:22.181548 7734 consensus_peers.cc:485] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d2a7bfaac904352a9c9c12c09b563f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:22.181829 11811 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.182248 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.183964 7664 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.185359 7926 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.186494 7271 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.185807 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240501 14:03:22.187510 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.188424 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.189567 11811 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.203725 11873 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.204524 7401 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.210083 9544 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.211795 6997 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:22.215297 7929 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.216409 7139 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.225512 7271 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.230325 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.245927 7797 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
W20240501 14:03:22.249365 7731 consensus_peers.cc:485] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64b9b1b64def483aaed2a03f7d356915. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:22.249589 9677 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.249709 7402 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.250320 11811 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.250603 11873 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.250927 7271 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.250953 7139 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.251569 11811 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.251849 11290 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.251965 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.252655 11873 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.260805 7664 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.261500 11751 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.262859 11751 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.265337 7402 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.265973 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.272122 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.274941 6998 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.275066 9678 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.278347 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.286150 7664 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:22.310461 7929 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
W20240501 14:03:22.314352 7865 consensus_peers.cc:485] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dc517698d5a4ae19525c96334ce7417. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:22.314707 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.314707 7140 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.315016 9544 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.315264 11741 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.315351 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.315341 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.315826 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:22.316282 11563 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.316538 11739 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.316761 11455 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.316999 11202 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.317261 11743 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:22.336143 11455 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.337296 7402 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.343886 6993 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.346349 9544 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.349378 7140 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.356894 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.370466 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.373514 7271 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:22.490967 7140 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.491565 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.574726 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.575284 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.606853 7402 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.607409 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.629051 9812 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.629714 11683 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.630367 7797 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.630899 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.630872 11683 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.631836 11683 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.641904 11960 ts_tablet_manager.cc:902] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:22.644752 7534 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:22.647233 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:22.678118 11960 tablet_copy_client.cc:308] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:22.678896 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet 0d2a7bfaac904352a9c9c12c09b563f5 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:49949)
I20240501 14:03:22.679122 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet 0d2a7bfaac904352a9c9c12c09b563f5 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:49949: session id = 41ab226fa6b7403b8b44642e9470517d-0d2a7bfaac904352a9c9c12c09b563f5
I20240501 14:03:22.681635 7818 tablet_copy_source_session.cc:205] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:22.682951 11960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2a7bfaac904352a9c9c12c09b563f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:22.688167 11960 tablet_copy_client.cc:783] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:22.688417 11960 tablet_copy_client.cc:647] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:22.690318 11960 tablet_copy_client.cc:523] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:22.692919 11960 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:22.732103 11960 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:22.732560 11960 tablet_bootstrap.cc:492] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:22.732872 11960 ts_tablet_manager.cc:1367] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.040s user 0.037s sys 0.000s
I20240501 14:03:22.733673 11960 raft_consensus.cc:348] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:22.734023 11960 raft_consensus.cc:729] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:22.734277 11960 consensus_queue.cc:260] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:22.734946 11960 ts_tablet_manager.cc:1398] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:03:22.735895 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-0d2a7bfaac904352a9c9c12c09b563f5 received from {username='slave'} at 127.22.5.201:49949
I20240501 14:03:22.736035 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-0d2a7bfaac904352a9c9c12c09b563f5 on tablet 0d2a7bfaac904352a9c9c12c09b563f5 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:22.765678 11960 ts_tablet_manager.cc:902] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:22.766889 11960 tablet_copy_client.cc:308] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:22.767529 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 2dc517698d5a4ae19525c96334ce7417 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:33315)
I20240501 14:03:22.767683 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 2dc517698d5a4ae19525c96334ce7417 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:33315: session id = 41ab226fa6b7403b8b44642e9470517d-2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:22.769711 7949 tablet_copy_source_session.cc:205] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:22.770953 11960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:22.775831 11960 tablet_copy_client.cc:783] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:22.776041 11960 tablet_copy_client.cc:647] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:22.777645 11960 tablet_copy_client.cc:523] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:22.780167 11960 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:22.818933 11960 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:22.819386 11960 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:22.819664 11960 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20240501 14:03:22.820484 11960 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:22.820722 11960 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:22.820969 11960 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:22.821655 11960 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:03:22.822504 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-2dc517698d5a4ae19525c96334ce7417 received from {username='slave'} at 127.22.5.201:33315
I20240501 14:03:22.822644 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-2dc517698d5a4ae19525c96334ce7417 on tablet 2dc517698d5a4ae19525c96334ce7417 with peer 41ab226fa6b7403b8b44642e9470517d
I20240501 14:03:22.852422 11999 ts_tablet_manager.cc:902] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:22.863639 11999 tablet_copy_client.cc:308] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:22.864658 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet 64b9b1b64def483aaed2a03f7d356915 from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:60165)
I20240501 14:03:22.864801 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet 64b9b1b64def483aaed2a03f7d356915 from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:60165: session id = cc3a47f051a546a5a7788d088f061dd5-64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:22.867216 7818 tablet_copy_source_session.cc:205] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:22.868368 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:22.875051 11999 tablet_copy_client.cc:783] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:22.875372 11999 tablet_copy_client.cc:647] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:22.878003 11999 tablet_copy_client.cc:523] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:22.880977 11999 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:22.923307 11999 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:22.923772 11999 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:22.924057 11999 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.043s user 0.033s sys 0.009s
I20240501 14:03:22.924811 11999 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:22.925186 11999 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:22.925400 11999 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:22.926353 11999 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
I20240501 14:03:22.927142 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-64b9b1b64def483aaed2a03f7d356915 received from {username='slave'} at 127.22.5.200:60165
I20240501 14:03:22.927282 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-64b9b1b64def483aaed2a03f7d356915 on tablet 64b9b1b64def483aaed2a03f7d356915 with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:23.142583 7272 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:23.145244 9677 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.145416 7797 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.145689 7534 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.145927 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.146474 11968 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.146795 11963 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.146847 7402 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.147805 9812 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.148584 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:03:23.149231 11963 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.149470 11968 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.149768 11773 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:23.152621 7797 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.152984 7929 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.153719 7402 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.158027 7000 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 2 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.158433 9812 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.151367 11968 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.168787 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2264386e494644aa84e8109c816aaefa with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:23.170377 9677 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.170477 7534 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:23.179631 9677 raft_consensus.cc:1203] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:23.183465 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2264386e494644aa84e8109c816aaefa with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:23.189337 12032 tablet_replica.cc:330] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:23.189575 12032 raft_consensus.cc:2229] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:23.189787 12032 raft_consensus.cc:2258] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:23.190661 12032 ts_tablet_manager.cc:1875] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:23.195380 12032 ts_tablet_manager.cc:1888] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 14:03:23.195520 12032 log.cc:1196] T 2264386e494644aa84e8109c816aaefa P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/2264386e494644aa84e8109c816aaefa
I20240501 14:03:23.196273 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 2264386e494644aa84e8109c816aaefa (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:23.247385 9677 raft_consensus.cc:1203] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:23.469199 9544 raft_consensus.cc:1203] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:23.602483 12030 raft_consensus.cc:1050] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:23.603402 12030 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:23.606616 9677 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.607432 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.608095 7140 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.608469 7272 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.609153 7402 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.609071 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.610306 9544 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.610579 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.610919 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.611474 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.612120 7929 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.612829 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.617503 7667 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.618264 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.623008 7402 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.621436 11873 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.630657 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.630909 9544 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.637188 6995 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.638231 9677 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.639871 7929 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.643288 7272 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.643568 7140 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.652822 7797 consensus_queue.cc:237] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:23.655527 9677 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.656158 7272 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.656183 7140 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.656809 7664 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.656778 12086 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.657301 11873 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.657532 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.658058 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.660985 9544 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.661682 7929 raft_consensus.cc:1261] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:23.661844 11751 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.661559 12086 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.662231 12085 consensus_queue.cc:1035] T 0d2a7bfaac904352a9c9c12c09b563f5 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:23.662890 9677 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.663497 7272 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.664124 7926 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.666034 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d2a7bfaac904352a9c9c12c09b563f5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:23.667033 7140 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.667701 7664 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.670250 6995 catalog_manager.cc:5484] T 0d2a7bfaac904352a9c9c12c09b563f5 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.672380 9544 raft_consensus.cc:2932] T 0d2a7bfaac904352a9c9c12c09b563f5 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.676064 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2a7bfaac904352a9c9c12c09b563f5 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:23.678414 12095 tablet_replica.cc:330] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:23.679097 12095 raft_consensus.cc:2229] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:23.679533 12095 raft_consensus.cc:2258] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:23.680531 12095 ts_tablet_manager.cc:1875] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:23.686201 12095 ts_tablet_manager.cc:1888] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:23.686307 12095 log.cc:1196] T 0d2a7bfaac904352a9c9c12c09b563f5 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d2a7bfaac904352a9c9c12c09b563f5
I20240501 14:03:23.687031 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet 0d2a7bfaac904352a9c9c12c09b563f5 (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
I20240501 14:03:23.832873 11743 raft_consensus.cc:1050] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:23.833834 11743 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:23.836587 9677 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.837046 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.837317 12124 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.838148 7140 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.845486 12125 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.845723 7272 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.846158 12125 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.846452 12124 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.848361 9544 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.849509 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.850276 12127 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.851015 12124 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.853083 12127 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.854018 7272 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.859373 6998 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.862552 9544 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.864277 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.882102 7402 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.882243 7140 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.885833 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:23.886798 9678 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:23.892083 12132 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:23.898851 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.004931 12030 raft_consensus.cc:1050] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER cc3a47f051a546a5a7788d088f061dd5 to VOTER
I20240501 14:03:24.006587 12030 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:24.011197 9544 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.011335 9678 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.011956 12086 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.012256 11873 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.013790 7140 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.013793 7402 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.014405 11751 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.014575 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.015213 12085 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.015569 12150 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.018944 7926 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.020047 11873 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.029577 7664 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:24.030850 12030 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:24.043236 12086 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.048717 9544 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.048482 7664 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.053792 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.054028 6995 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.057046 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.065153 7402 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.069267 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.074019 7795 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } }
I20240501 14:03:24.073393 9678 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.081609 9544 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.082227 12086 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.082743 7271 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.083035 7664 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.083734 9677 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.084766 12086 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.086812 7140 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.087090 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.087625 12150 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.088172 12150 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.091400 11873 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.091943 7139 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.093277 11873 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.096742 6999 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.097757 12085 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.099853 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.102381 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64b9b1b64def483aaed2a03f7d356915 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:24.103152 9544 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.103416 7664 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.103593 9677 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.106962 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } }
I20240501 14:03:24.110854 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b9b1b64def483aaed2a03f7d356915 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:24.111297 12095 tablet_replica.cc:330] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:24.111500 12095 raft_consensus.cc:2229] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:24.112167 12095 raft_consensus.cc:2258] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:24.113246 12095 ts_tablet_manager.cc:1875] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:24.122553 12095 ts_tablet_manager.cc:1888] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:24.122727 12095 log.cc:1196] T 64b9b1b64def483aaed2a03f7d356915 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:24.128242 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet 64b9b1b64def483aaed2a03f7d356915 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:24.504387 7926 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:24.507685 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.508227 9677 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.508543 12178 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.508805 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.508853 9544 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.509528 12178 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.509699 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.509801 12132 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.510262 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:24.510306 12128 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.511461 12128 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.511788 12132 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:24.515369 12178 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.516018 9544 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.516104 7272 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.516834 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.519789 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dc517698d5a4ae19525c96334ce7417 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:24.520342 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.521536 9677 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.526381 6998 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.529374 7402 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:24.531822 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc517698d5a4ae19525c96334ce7417 with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:24.537904 12180 tablet_replica.cc:330] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:24.538295 12180 raft_consensus.cc:2229] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:24.538697 12180 raft_consensus.cc:2258] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:24.539710 12180 ts_tablet_manager.cc:1875] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:24.546099 12180 ts_tablet_manager.cc:1888] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:24.546200 12180 log.cc:1196] T 2dc517698d5a4ae19525c96334ce7417 P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:24.546886 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 2dc517698d5a4ae19525c96334ce7417 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:24.822917 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:24.828259 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:24.837894 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.866604 7120 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:24.897194 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:24.917631 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.976433 7251 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:25.001885 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:25.019621 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:25.068782 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:25.321619 7120 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:25.333058 9658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:25.337297 7515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:25.356225 9794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:25.373754 9524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:25.375555 7251 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:25.388242 7907 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.423027 7776 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:25.467839 7385 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:25.510393 7340 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 114321us
I20240501 14:03:25.580272 7647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:25.735797 7271 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:25.760002 7926 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:25.766968 9677 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 14:03:25.767110 7862 consensus_peers.cc:485] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d -> Peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Couldn't send request to peer ceb055f6a1a540dcb61e1e11a157cdb8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:25.767843 9811 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.767935 7797 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.768417 7140 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.768613 12178 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.769078 7664 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.769280 12178 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.769938 12178 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.774768 12132 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.775558 12132 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.777586 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.778368 12132 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.815032 12178 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.818176 7797 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.820501 7667 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.826193 9677 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.834446 6995 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.853587 9811 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.864154 7140 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.874310 7272 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:25.876139 7402 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:03:25.876787 7209 consensus_peers.cc:485] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc -> Peer 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): Couldn't send request to peer 41ab226fa6b7403b8b44642e9470517d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab9f466bd0244debff3e228ba99eb51. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:25.876988 12028 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:25.888208 7667 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
W20240501 14:03:25.907282 7601 consensus_peers.cc:485] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab -> Peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Couldn't send request to peer 704fb2d37343406790224dc76b40f210. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:25.909397 7140 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.909770 7272 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.910915 12148 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.913554 9544 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.915197 12148 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.915506 7926 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.916074 12338 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.916792 12338 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.925495 7797 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.926645 9811 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:25.930680 12148 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.931591 12148 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:25.938915 7926 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:25.940358 12028 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:25.947705 9811 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:25.945072 12338 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:25.950057 12028 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:25.951257 7272 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:25.954736 7140 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:25.956526 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:25.970944 7273 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:25.975276 7139 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:25.983230 6998 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:25.991675 9814 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
W20240501 14:03:26.000380 7206 consensus_peers.cc:485] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc -> Peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): Couldn't send request to peer ceb055f6a1a540dcb61e1e11a157cdb8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.001881 9677 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.002936 7667 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:26.002985 7926 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.003723 7139 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.003832 7795 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:26.004303 12028 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.004472 7534 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.004592 12344 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:26.005048 9544 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.005306 12333 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.007613 7795 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.008077 12028 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.008769 12028 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:26.009387 12028 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.009922 12333 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.014982 7797 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.034004 12344 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.042181 12333 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.042827 7140 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
W20240501 14:03:26.047581 12346 log.cc:924] Time spent T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20240501 14:03:26.054919 7206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.198:45423, user_credentials={real_user=slave}} blocked reactor thread for 46816.3us
I20240501 14:03:26.056030 6993 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.065433 9812 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.078219 7798 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.082011 12344 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.083700 7139 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.088811 7926 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.099236 6998 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.108250 7798 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.112354 7534 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
W20240501 14:03:26.134754 7209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.201:33735, user_credentials={real_user=slave}} blocked reactor thread for 127798us
I20240501 14:03:26.148090 7664 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.151010 9543 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.156960 9677 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } } }
I20240501 14:03:26.169767 7926 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.169920 7402 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } } }
I20240501 14:03:26.174543 7929 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.194725 9543 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.217295 7926 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } }
I20240501 14:03:26.225399 7929 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } }
W20240501 14:03:26.230772 7864 consensus_peers.cc:485] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d -> Peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): Couldn't send request to peer 0d05240051574799acbbe13e21b262b4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.231554 9543 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.232529 7798 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.233500 9812 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.234587 7402 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.235514 7271 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.236137 12322 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.236929 12323 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.237520 12323 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.239567 7667 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.250110 12323 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.250484 12315 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.251250 12323 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.254505 9677 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.255165 7139 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.255141 12323 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.255728 12315 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.260366 7534 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.261205 12323 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.290335 9812 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.292426 7271 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.292738 7797 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 14:03:26.293764 7862 consensus_peers.cc:485] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d -> Peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Couldn't send request to peer db6b830d9bbe4451bd349c6ad6e14bab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.293906 12397 ts_tablet_manager.cc:902] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:26.294288 12178 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.295022 7271 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.297772 12322 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.298538 12178 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.294289 12315 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.307217 6998 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.319486 7929 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:26.353506 12178 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.364696 6998 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:26.368248 7139 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.367533 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.369751 7667 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.370512 9814 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.375576 7271 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.380717 9677 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.387547 9678 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.384464 7797 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.389565 12315 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.381630 7798 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.409623 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.412184 9544 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.413476 9812 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:26.414139 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.416549 12398 ts_tablet_manager.cc:902] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Initiating tablet copy from peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:26.416903 12316 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.421635 12316 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:03:26.424005 7862 consensus_peers.cc:485] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dc517698d5a4ae19525c96334ce7417. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.426383 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.426839 12316 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.433621 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.434517 7402 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.439566 12414 ts_tablet_manager.cc:902] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:26.447125 12316 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.449493 12315 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.461943 9677 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.463634 9543 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } } }
I20240501 14:03:26.468434 6998 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.470350 9543 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.470993 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.485960 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.498723 12414 tablet_copy_client.cc:308] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:26.499667 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 1ab9f466bd0244debff3e228ba99eb51 from peer 41ab226fa6b7403b8b44642e9470517d ({username='slave'} at 127.22.5.201:38919)
I20240501 14:03:26.499855 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 1ab9f466bd0244debff3e228ba99eb51 from peer 41ab226fa6b7403b8b44642e9470517d at {username='slave'} at 127.22.5.201:38919: session id = 41ab226fa6b7403b8b44642e9470517d-1ab9f466bd0244debff3e228ba99eb51
I20240501 14:03:26.502557 7271 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.503794 7293 tablet_copy_source_session.cc:205] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.512753 12397 tablet_copy_client.cc:308] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:26.513743 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet c5396b2071184070bcb14763e1df7b5f from peer ceb055f6a1a540dcb61e1e11a157cdb8 ({username='slave'} at 127.22.5.195:49617)
I20240501 14:03:26.513907 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet c5396b2071184070bcb14763e1df7b5f from peer ceb055f6a1a540dcb61e1e11a157cdb8 at {username='slave'} at 127.22.5.195:49617: session id = ceb055f6a1a540dcb61e1e11a157cdb8-c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:26.516805 7949 tablet_copy_source_session.cc:205] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.518440 12397 ts_tablet_manager.cc:1875] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:26.519877 12414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab9f466bd0244debff3e228ba99eb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.526554 7667 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.535442 7797 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.539561 12414 tablet_copy_client.cc:783] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:26.539965 12414 tablet_copy_client.cc:647] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:26.541965 12414 tablet_copy_client.cc:523] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:26.545424 12414 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap starting.
I20240501 14:03:26.557435 7797 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:26.560092 12398 tablet_copy_client.cc:308] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:42421
I20240501 14:03:26.562996 7687 tablet_copy_service.cc:140] P db6b830d9bbe4451bd349c6ad6e14bab: Received BeginTabletCopySession request for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer 704fb2d37343406790224dc76b40f210 ({username='slave'} at 127.22.5.196:33841)
I20240501 14:03:26.563213 7687 tablet_copy_service.cc:161] P db6b830d9bbe4451bd349c6ad6e14bab: Beginning new tablet copy session on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 from peer 704fb2d37343406790224dc76b40f210 at {username='slave'} at 127.22.5.196:33841: session id = 704fb2d37343406790224dc76b40f210-4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:26.566185 7687 tablet_copy_source_session.cc:205] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.568212 9543 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.568754 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.569598 7139 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.570233 9677 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.570991 7667 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.574116 12397 ts_tablet_manager.cc:1888] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 14:03:26.574805 12397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5396b2071184070bcb14763e1df7b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.577898 12397 tablet_copy_client.cc:783] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:26.578202 12397 tablet_copy_client.cc:647] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:26.579159 7271 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.580466 12397 tablet_copy_client.cc:523] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:26.582280 12398 ts_tablet_manager.cc:1875] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:26.584830 12397 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:03:26.586685 7139 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:26.598964 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20240501 14:03:26.603188 12030 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20240501 14:03:26.611934 12446 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 14:03:26.613493 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.638420 12445 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:03:26.640245 7731 consensus_peers.cc:485] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64b9b1b64def483aaed2a03f7d356915. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.640703 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.648914 12443 ts_tablet_manager.cc:902] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
W20240501 14:03:26.651650 7733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.193:41299, user_credentials={real_user=slave}} blocked reactor thread for 51773.7us
I20240501 14:03:26.652400 12294 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.662735 7000 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 reported cstate change: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:26.666471 7667 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.676168 12398 ts_tablet_manager.cc:1888] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:26.675873 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.676549 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e104d44c5fa4a29be94a33cf7e93ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.677757 9677 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.683111 12460 ts_tablet_manager.cc:902] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:26.684329 9543 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.690666 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.695302 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:26.698422 12398 tablet_copy_client.cc:783] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:26.698771 12398 tablet_copy_client.cc:647] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:26.702404 12398 tablet_copy_client.cc:523] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:26.707978 12398 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
W20240501 14:03:26.716253 7075 consensus_peers.cc:485] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 -> Peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Couldn't send request to peer 704fb2d37343406790224dc76b40f210. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.717716 9677 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.717747 12460 tablet_copy_client.cc:308] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:26.718364 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.718565 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet bd9f7d11351b4f8d84021720488abf25 from peer db6b830d9bbe4451bd349c6ad6e14bab ({username='slave'} at 127.22.5.197:60109)
I20240501 14:03:26.718724 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet bd9f7d11351b4f8d84021720488abf25 from peer db6b830d9bbe4451bd349c6ad6e14bab at {username='slave'} at 127.22.5.197:60109: session id = db6b830d9bbe4451bd349c6ad6e14bab-bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:26.721871 7949 tablet_copy_source_session.cc:205] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.724035 12460 ts_tablet_manager.cc:1875] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:26.729761 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.730475 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.743830 12460 ts_tablet_manager.cc:1888] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:26.744307 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f7d11351b4f8d84021720488abf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.746840 7797 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.747524 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.748114 12460 tablet_copy_client.cc:783] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:26.748411 12460 tablet_copy_client.cc:647] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:26.750505 12460 tablet_copy_client.cc:523] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:26.759485 12460 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:03:26.765062 12469 ts_tablet_manager.cc:902] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:26.768455 7929 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.769515 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.785594 9812 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.787338 11526 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.796391 12398 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:26.796921 12398 tablet_bootstrap.cc:492] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Bootstrap complete.
I20240501 14:03:26.797348 12398 ts_tablet_manager.cc:1367] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.089s user 0.068s sys 0.006s
I20240501 14:03:26.799623 12398 raft_consensus.cc:348] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:26.800186 12398 raft_consensus.cc:729] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: LEARNER
I20240501 14:03:26.800519 12398 consensus_queue.cc:260] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:26.804649 12398 ts_tablet_manager.cc:1398] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:03:26.807871 7687 tablet_copy_service.cc:342] P db6b830d9bbe4451bd349c6ad6e14bab: Request end of tablet copy session 704fb2d37343406790224dc76b40f210-4e104d44c5fa4a29be94a33cf7e93ba6 received from {username='slave'} at 127.22.5.196:33841
I20240501 14:03:26.808015 7687 tablet_copy_service.cc:433] P db6b830d9bbe4451bd349c6ad6e14bab: ending tablet copy session 704fb2d37343406790224dc76b40f210-4e104d44c5fa4a29be94a33cf7e93ba6 on tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with peer 704fb2d37343406790224dc76b40f210
I20240501 14:03:26.832842 12414 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:26.835840 12414 tablet_bootstrap.cc:492] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Bootstrap complete.
I20240501 14:03:26.836171 12414 ts_tablet_manager.cc:1367] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Time spent bootstrapping tablet: real 0.291s user 0.050s sys 0.008s
I20240501 14:03:26.836995 12414 raft_consensus.cc:348] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:26.837388 12414 raft_consensus.cc:729] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41ab226fa6b7403b8b44642e9470517d, State: Initialized, Role: LEARNER
I20240501 14:03:26.837625 12414 consensus_queue.cc:260] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: NON_VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: true } }
I20240501 14:03:26.838424 12414 ts_tablet_manager.cc:1398] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240501 14:03:26.841385 7074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.202:37399, user_credentials={real_user=slave}} blocked reactor thread for 55091.7us
I20240501 14:03:26.841907 7293 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session 41ab226fa6b7403b8b44642e9470517d-1ab9f466bd0244debff3e228ba99eb51 received from {username='slave'} at 127.22.5.201:38919
I20240501 14:03:26.842054 7293 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session 41ab226fa6b7403b8b44642e9470517d-1ab9f466bd0244debff3e228ba99eb51 on tablet 1ab9f466bd0244debff3e228ba99eb51 with peer 41ab226fa6b7403b8b44642e9470517d
W20240501 14:03:26.841681 7075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.199:33993, user_credentials={real_user=slave}} blocked reactor thread for 55125.6us
I20240501 14:03:26.843079 7667 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:26.846709 12474 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.856357 12475 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:26.857761 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.860110 12485 ts_tablet_manager.cc:902] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993)
I20240501 14:03:26.863409 7797 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.866575 12469 tablet_copy_client.cc:308] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:26.869743 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer 0d05240051574799acbbe13e21b262b4 ({username='slave'} at 127.22.5.193:60433)
I20240501 14:03:26.869921 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 from peer 0d05240051574799acbbe13e21b262b4 at {username='slave'} at 127.22.5.193:60433: session id = 0d05240051574799acbbe13e21b262b4-1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:26.875764 7949 tablet_copy_source_session.cc:205] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.877266 6993 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.877617 7667 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.891122 12469 ts_tablet_manager.cc:1875] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:26.902833 12485 tablet_copy_client.cc:308] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.199:33993
I20240501 14:03:26.903690 7949 tablet_copy_service.cc:140] P cc7e7386a9e540eabc4648e147073f1d: Received BeginTabletCopySession request for tablet 2dc517698d5a4ae19525c96334ce7417 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:60515)
I20240501 14:03:26.903884 7949 tablet_copy_service.cc:161] P cc7e7386a9e540eabc4648e147073f1d: Beginning new tablet copy session on tablet 2dc517698d5a4ae19525c96334ce7417 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:60515: session id = 1c61c125cd144590ae1e54a2f87d9187-2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:26.907819 7949 tablet_copy_source_session.cc:205] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.909647 12485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc517698d5a4ae19525c96334ce7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.916186 12443 tablet_copy_client.cc:308] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:26.917069 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 253a0abd26814fb29f9adede664c5988 from peer ceb055f6a1a540dcb61e1e11a157cdb8 ({username='slave'} at 127.22.5.195:35853)
I20240501 14:03:26.917286 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 253a0abd26814fb29f9adede664c5988 from peer ceb055f6a1a540dcb61e1e11a157cdb8 at {username='slave'} at 127.22.5.195:35853: session id = ceb055f6a1a540dcb61e1e11a157cdb8-253a0abd26814fb29f9adede664c5988
I20240501 14:03:26.920022 7293 tablet_copy_source_session.cc:205] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:26.922178 7926 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.922227 9812 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.924075 12485 tablet_copy_client.cc:783] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:26.924342 12485 tablet_copy_client.cc:647] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:26.926287 12485 tablet_copy_client.cc:523] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:26.930357 12485 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:26.931082 9678 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:26.942333 12397 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:26.946664 12469 ts_tablet_manager.cc:1888] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 14:03:26.950815 12397 tablet_bootstrap.cc:492] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap complete.
I20240501 14:03:26.953576 12397 ts_tablet_manager.cc:1367] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.369s user 0.094s sys 0.060s
I20240501 14:03:26.957759 12469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab7d2ddba3440f0a1311f9eda7d3930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:26.960739 12397 raft_consensus.cc:348] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:26.962491 12397 raft_consensus.cc:729] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: LEARNER
I20240501 14:03:26.963863 12397 consensus_queue.cc:260] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:26.975229 12397 ts_tablet_manager.cc:1398] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:03:26.976161 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session ceb055f6a1a540dcb61e1e11a157cdb8-c5396b2071184070bcb14763e1df7b5f received from {username='slave'} at 127.22.5.195:49617
I20240501 14:03:26.976297 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session ceb055f6a1a540dcb61e1e11a157cdb8-c5396b2071184070bcb14763e1df7b5f on tablet c5396b2071184070bcb14763e1df7b5f with peer ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:03:26.992133 9678 raft_consensus.cc:1203] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:27.001438 12443 ts_tablet_manager.cc:1875] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:27.003701 12469 tablet_copy_client.cc:783] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.004164 12469 tablet_copy_client.cc:647] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.015604 12469 tablet_copy_client.cc:523] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.029026 12443 ts_tablet_manager.cc:1888] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 14:03:27.033285 12469 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Bootstrap starting.
I20240501 14:03:27.049587 7139 consensus_queue.cc:237] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:27.055388 12443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253a0abd26814fb29f9adede664c5988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:27.071286 12443 tablet_copy_client.cc:783] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.073518 12443 tablet_copy_client.cc:647] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.075057 9812 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.075726 7271 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.076072 12481 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.076252 7926 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.076855 11526 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.074683 7401 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.077500 12475 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.077872 11526 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.091917 7535 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.092717 11526 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.095429 11526 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.096856 7797 raft_consensus.cc:1261] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.098142 12443 tablet_copy_client.cc:523] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.099331 12481 consensus_queue.cc:1035] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.099357 7271 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.105728 7000 catalog_manager.cc:5484] T 11da808a7f774e5ea7bd7ef298f1180e P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
W20240501 14:03:27.108592 7074 consensus_peers.cc:485] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4 -> Peer cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): Couldn't send request to peer cc3a47f051a546a5a7788d088f061dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11da808a7f774e5ea7bd7ef298f1180e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:27.109501 7535 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.113982 12443 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap starting.
I20240501 14:03:27.113517 7797 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.116925 9812 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.123185 7401 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.126276 7926 raft_consensus.cc:2932] T 11da808a7f774e5ea7bd7ef298f1180e P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } } }
I20240501 14:03:27.181360 7667 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } }
I20240501 14:03:27.205651 9678 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.206516 7139 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.210043 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.210697 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:27.215301 7600 consensus_peers.cc:485] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab -> Peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): Couldn't send request to peer c8f737f3f6994db297b47fb23dad1b13. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:27.218699 7535 raft_consensus.cc:1203] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.219264 12485 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.219705 12485 tablet_bootstrap.cc:492] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:27.220005 12485 ts_tablet_manager.cc:1367] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.290s user 0.076s sys 0.050s
I20240501 14:03:27.221441 12527 ts_tablet_manager.cc:902] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423)
I20240501 14:03:27.224614 7401 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.225368 12470 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.227581 12485 raft_consensus.cc:348] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:27.227907 12485 raft_consensus.cc:729] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:27.228106 12485 consensus_queue.cc:260] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:27.229321 7271 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.230290 12470 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.232498 12485 ts_tablet_manager.cc:1398] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20240501 14:03:27.235100 7926 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.235569 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-2dc517698d5a4ae19525c96334ce7417 received from {username='slave'} at 127.22.5.202:60515
I20240501 14:03:27.235694 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-2dc517698d5a4ae19525c96334ce7417 on tablet 2dc517698d5a4ae19525c96334ce7417 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:27.235800 12470 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.236017 7402 raft_consensus.cc:1203] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.246050 7535 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:27.246702 12470 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:27.248483 12443 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.249101 12443 tablet_bootstrap.cc:492] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Bootstrap complete.
I20240501 14:03:27.249508 12443 ts_tablet_manager.cc:1367] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent bootstrapping tablet: real 0.136s user 0.050s sys 0.012s
I20240501 14:03:27.250581 12443 raft_consensus.cc:348] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:27.251070 12443 raft_consensus.cc:729] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ceb055f6a1a540dcb61e1e11a157cdb8, State: Initialized, Role: LEARNER
I20240501 14:03:27.251355 12443 consensus_queue.cc:260] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: NON_VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: true } }
I20240501 14:03:27.256335 12527 tablet_copy_client.cc:308] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.198:45423
I20240501 14:03:27.261644 12443 ts_tablet_manager.cc:1398] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240501 14:03:27.262503 7293 tablet_copy_service.cc:342] P b3b866abc97f437e8f24eccb653b3dcc: Request end of tablet copy session ceb055f6a1a540dcb61e1e11a157cdb8-253a0abd26814fb29f9adede664c5988 received from {username='slave'} at 127.22.5.195:35853
I20240501 14:03:27.262635 7293 tablet_copy_service.cc:433] P b3b866abc97f437e8f24eccb653b3dcc: ending tablet copy session ceb055f6a1a540dcb61e1e11a157cdb8-253a0abd26814fb29f9adede664c5988 on tablet 253a0abd26814fb29f9adede664c5988 with peer ceb055f6a1a540dcb61e1e11a157cdb8
I20240501 14:03:27.262729 7401 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:27.263571 7818 tablet_copy_service.cc:140] P c8f737f3f6994db297b47fb23dad1b13: Received BeginTabletCopySession request for tablet 64b9b1b64def483aaed2a03f7d356915 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:42055)
I20240501 14:03:27.263748 7818 tablet_copy_service.cc:161] P c8f737f3f6994db297b47fb23dad1b13: Beginning new tablet copy session on tablet 64b9b1b64def483aaed2a03f7d356915 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:42055: session id = 1c61c125cd144590ae1e54a2f87d9187-64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:27.266767 7818 tablet_copy_source_session.cc:205] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:27.268940 12470 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.270890 7535 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.278295 7402 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.278374 6993 catalog_manager.cc:5484] T 210980fcec0c4358943ddd2cc7e935a6 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.280314 12527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b9b1b64def483aaed2a03f7d356915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:27.293231 7271 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.302551 7271 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.302696 7929 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.303627 9677 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.303668 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.304299 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.304773 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.306681 12398 ts_tablet_manager.cc:902] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:27.307698 12527 tablet_copy_client.cc:783] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.308022 12527 tablet_copy_client.cc:647] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.309919 7140 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.319029 9812 raft_consensus.cc:1203] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.324252 12527 tablet_copy_client.cc:523] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.322710 9678 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.324992 7926 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } } }
I20240501 14:03:27.334353 12527 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
W20240501 14:03:27.369396 12543 log.cc:924] Time spent T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240501 14:03:27.373986 12398 tablet_copy_client.cc:308] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:27.374855 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 704fb2d37343406790224dc76b40f210 ({username='slave'} at 127.22.5.196:54539)
I20240501 14:03:27.375056 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet b227b60efb7343bba8c4e89f81c3f6ff from peer 704fb2d37343406790224dc76b40f210 at {username='slave'} at 127.22.5.196:54539: session id = 704fb2d37343406790224dc76b40f210-b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:27.375617 12460 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.378228 7161 tablet_copy_source_session.cc:205] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:27.378307 12460 tablet_bootstrap.cc:492] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap complete.
I20240501 14:03:27.378615 12460 ts_tablet_manager.cc:1367] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent bootstrapping tablet: real 0.619s user 0.070s sys 0.009s
I20240501 14:03:27.380074 9543 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.380514 7797 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.381330 7667 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:03:27.381484 12460 raft_consensus.cc:348] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } }
W20240501 14:03:27.383033 7338 consensus_peers.cc:485] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cf39e6be52426c93c1534754be7264. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:27.383258 12460 raft_consensus.cc:729] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db6b830d9bbe4451bd349c6ad6e14bab, State: Initialized, Role: LEARNER
I20240501 14:03:27.383332 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.383656 12460 consensus_queue.cc:260] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } }
I20240501 14:03:27.383915 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.384501 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:03:27.399947 12460 ts_tablet_manager.cc:1398] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240501 14:03:27.402422 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session db6b830d9bbe4451bd349c6ad6e14bab-bd9f7d11351b4f8d84021720488abf25 received from {username='slave'} at 127.22.5.197:60109
I20240501 14:03:27.402554 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session db6b830d9bbe4451bd349c6ad6e14bab-bd9f7d11351b4f8d84021720488abf25 on tablet bd9f7d11351b4f8d84021720488abf25 with peer db6b830d9bbe4451bd349c6ad6e14bab
I20240501 14:03:27.403841 12398 ts_tablet_manager.cc:1875] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:27.405141 12472 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.411593 7664 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.411593 9544 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.419718 6993 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.428887 9677 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.429245 7929 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.462744 7272 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.473309 7797 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } }
I20240501 14:03:27.476342 12469 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.476877 12469 tablet_bootstrap.cc:492] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Bootstrap complete.
I20240501 14:03:27.477218 12469 ts_tablet_manager.cc:1367] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Time spent bootstrapping tablet: real 0.444s user 0.072s sys 0.000s
I20240501 14:03:27.478149 12469 raft_consensus.cc:348] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } }
I20240501 14:03:27.478510 12469 raft_consensus.cc:729] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d05240051574799acbbe13e21b262b4, State: Initialized, Role: LEARNER
I20240501 14:03:27.478762 12469 consensus_queue.cc:260] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: NON_VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: true } }
I20240501 14:03:27.480201 12353 raft_consensus.cc:1050] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER 41ab226fa6b7403b8b44642e9470517d to VOTER
I20240501 14:03:27.481468 12353 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:27.485750 12469 ts_tablet_manager.cc:1398] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:03:27.486508 7949 tablet_copy_service.cc:342] P cc7e7386a9e540eabc4648e147073f1d: Request end of tablet copy session 0d05240051574799acbbe13e21b262b4-1ab7d2ddba3440f0a1311f9eda7d3930 received from {username='slave'} at 127.22.5.193:60433
I20240501 14:03:27.486627 7949 tablet_copy_service.cc:433] P cc7e7386a9e540eabc4648e147073f1d: ending tablet copy session 0d05240051574799acbbe13e21b262b4-1ab7d2ddba3440f0a1311f9eda7d3930 on tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with peer 0d05240051574799acbbe13e21b262b4
I20240501 14:03:27.497694 9677 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.498785 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.502584 12398 ts_tablet_manager.cc:1888] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:27.502923 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227b60efb7343bba8c4e89f81c3f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:27.506879 7667 raft_consensus.cc:1203] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.514891 12398 tablet_copy_client.cc:783] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.517426 12398 tablet_copy_client.cc:647] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.518675 12527 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.519706 12398 tablet_copy_client.cc:523] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.519831 12527 tablet_bootstrap.cc:492] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:27.520149 12527 ts_tablet_manager.cc:1367] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.186s user 0.060s sys 0.005s
I20240501 14:03:27.520980 12527 raft_consensus.cc:348] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:27.521330 12527 raft_consensus.cc:729] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:27.521317 7401 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.521901 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.521584 12527 consensus_queue.cc:260] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } }
I20240501 14:03:27.527447 12398 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:27.528163 12527 ts_tablet_manager.cc:1398] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:03:27.532518 7139 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.532904 7818 tablet_copy_service.cc:342] P c8f737f3f6994db297b47fb23dad1b13: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-64b9b1b64def483aaed2a03f7d356915 received from {username='slave'} at 127.22.5.202:42055
I20240501 14:03:27.533038 7818 tablet_copy_service.cc:433] P c8f737f3f6994db297b47fb23dad1b13: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-64b9b1b64def483aaed2a03f7d356915 on tablet 64b9b1b64def483aaed2a03f7d356915 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:27.533934 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.543908 7929 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.544833 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.587394 12358 raft_consensus.cc:1050] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab: attempting to promote NON_VOTER 704fb2d37343406790224dc76b40f210 to VOTER
I20240501 14:03:27.589094 12358 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } }
I20240501 14:03:27.596174 7667 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.596210 7401 raft_consensus.cc:1203] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.596333 9812 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.596802 12579 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.599174 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240501 14:03:27.601428 7797 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:27.602921 12579 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:27.603782 12579 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.604343 7401 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:27.610653 7272 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:03:27.611196 7339 consensus_peers.cc:485] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Couldn't send request to peer 704fb2d37343406790224dc76b40f210. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:27.612453 12563 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.613514 9678 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.615873 7535 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.616313 9814 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.617676 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.621078 12593 ts_tablet_manager.cc:902] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Initiating tablet copy from peer 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299)
I20240501 14:03:27.621451 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.623805 7402 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.635416 7271 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.635699 7140 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.636000 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.636466 12530 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.661574 9544 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.662163 7795 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.663735 7664 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:03:27.664760 7338 consensus_peers.cc:485] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): Couldn't send request to peer 1c61c125cd144590ae1e54a2f87d9187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cf39e6be52426c93c1534754be7264. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:27.665187 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.665891 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.666442 12396 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.669692 7926 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.670251 12530 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.677529 9544 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.678223 7795 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.678155 12530 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.679587 12549 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.680423 12358 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.685976 7664 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.686116 9544 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.694717 7000 catalog_manager.cc:5484] T 60cf39e6be52426c93c1534754be7264 P cc3a47f051a546a5a7788d088f061dd5 reported cstate change: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.701375 12315 raft_consensus.cc:1050] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 to VOTER
I20240501 14:03:27.703101 12315 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } }
I20240501 14:03:27.704923 12604 ts_tablet_manager.cc:902] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Initiating tablet copy from peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421)
I20240501 14:03:27.709844 12530 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.711266 9814 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.691574 6998 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 reported cstate change: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.716320 7272 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.719947 9544 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.723520 6993 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:27.729593 7535 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.734889 7927 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.734992 9812 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.749336 7666 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.753285 9813 raft_consensus.cc:1203] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.761169 7137 raft_consensus.cc:1203] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:27.769289 7926 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.786516 12604 tablet_copy_client.cc:308] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.197:42421
I20240501 14:03:27.787638 7687 tablet_copy_service.cc:140] P db6b830d9bbe4451bd349c6ad6e14bab: Received BeginTabletCopySession request for tablet 210980fcec0c4358943ddd2cc7e935a6 from peer c8f737f3f6994db297b47fb23dad1b13 ({username='slave'} at 127.22.5.198:60205)
I20240501 14:03:27.787853 7687 tablet_copy_service.cc:161] P db6b830d9bbe4451bd349c6ad6e14bab: Beginning new tablet copy session on tablet 210980fcec0c4358943ddd2cc7e935a6 from peer c8f737f3f6994db297b47fb23dad1b13 at {username='slave'} at 127.22.5.198:60205: session id = c8f737f3f6994db297b47fb23dad1b13-210980fcec0c4358943ddd2cc7e935a6
I20240501 14:03:27.792388 7687 tablet_copy_source_session.cc:205] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:27.794520 12604 ts_tablet_manager.cc:1875] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:27.805881 7795 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.808032 7271 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.822842 12604 ts_tablet_manager.cc:1888] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:27.823938 12132 raft_consensus.cc:1050] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:27.825935 12132 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:27.827795 12604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210980fcec0c4358943ddd2cc7e935a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:27.836983 12604 tablet_copy_client.cc:783] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.842231 7929 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.842870 12472 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.843056 12604 tablet_copy_client.cc:647] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.845034 7139 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.849516 9677 raft_consensus.cc:1261] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.849933 12593 tablet_copy_client.cc:308] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.193:41299
I20240501 14:03:27.850045 12604 tablet_copy_client.cc:523] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.850152 12472 consensus_queue.cc:1035] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.863094 9677 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.858242 7929 raft_consensus.cc:2932] T 60cf39e6be52426c93c1534754be7264 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:27.874939 12604 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap starting.
I20240501 14:03:27.882854 7402 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.883514 12323 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.899252 9675 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.899952 9676 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.900635 7795 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:27.907667 12631 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20240501 14:03:27.912642 12631 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.922729 7795 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.923910 7271 consensus_queue.cc:237] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } }
I20240501 14:03:27.928104 7401 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:27.928853 7139 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:27.928876 9676 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:27.928973 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:27.929558 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:27.929982 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:27.930538 7926 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:27.931181 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:27.939565 9812 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.943044 12631 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.943531 12589 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.944895 9676 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.950958 7797 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:27.951270 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab9f466bd0244debff3e228ba99eb51 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:27.951917 12592 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:27.952124 7139 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.953622 7926 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.961016 6997 catalog_manager.cc:5484] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:27.966732 7401 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.969879 7402 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:27.971230 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:27.974884 7795 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:27.988401 12631 raft_consensus.cc:1050] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab to VOTER
I20240501 14:03:27.990334 12631 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } }
I20240501 14:03:27.995697 7664 raft_consensus.cc:1261] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:03:28.018245 12398 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:28.018864 12398 tablet_bootstrap.cc:492] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Bootstrap complete.
I20240501 14:03:28.019387 12398 ts_tablet_manager.cc:1367] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.492s user 0.070s sys 0.007s
I20240501 14:03:28.020463 12398 raft_consensus.cc:348] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:28.020967 12398 raft_consensus.cc:729] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: LEARNER
I20240501 14:03:28.021301 12398 consensus_queue.cc:260] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:28.023011 9677 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.024219 7667 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.024506 9814 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.024662 12333 consensus_queue.cc:1035] T 1ab9f466bd0244debff3e228ba99eb51 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:28.024968 7271 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.025704 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.026520 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.027083 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.027544 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.027032 7667 raft_consensus.cc:2932] T 1ab9f466bd0244debff3e228ba99eb51 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } }
I20240501 14:03:28.032603 12582 raft_consensus.cc:1050] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc: attempting to promote NON_VOTER ceb055f6a1a540dcb61e1e11a157cdb8 to VOTER
I20240501 14:03:28.034150 12582 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } }
I20240501 14:03:28.036141 7139 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.036844 7534 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.037390 7795 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.037510 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.037887 12323 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.040021 12323 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.046412 7137 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.058055 12398 ts_tablet_manager.cc:1398] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240501 14:03:28.058770 7161 tablet_copy_service.cc:140] P 0d05240051574799acbbe13e21b262b4: Received BeginTabletCopySession request for tablet 11da808a7f774e5ea7bd7ef298f1180e from peer cc3a47f051a546a5a7788d088f061dd5 ({username='slave'} at 127.22.5.200:38709)
I20240501 14:03:28.058967 7161 tablet_copy_service.cc:161] P 0d05240051574799acbbe13e21b262b4: Beginning new tablet copy session on tablet 11da808a7f774e5ea7bd7ef298f1180e from peer cc3a47f051a546a5a7788d088f061dd5 at {username='slave'} at 127.22.5.200:38709: session id = cc3a47f051a546a5a7788d088f061dd5-11da808a7f774e5ea7bd7ef298f1180e
I20240501 14:03:28.061286 7160 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session 704fb2d37343406790224dc76b40f210-b227b60efb7343bba8c4e89f81c3f6ff received from {username='slave'} at 127.22.5.196:54539
I20240501 14:03:28.061425 7160 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session 704fb2d37343406790224dc76b40f210-b227b60efb7343bba8c4e89f81c3f6ff on tablet b227b60efb7343bba8c4e89f81c3f6ff with peer 704fb2d37343406790224dc76b40f210
I20240501 14:03:28.064081 7161 tablet_copy_source_session.cc:205] T 11da808a7f774e5ea7bd7ef298f1180e P 0d05240051574799acbbe13e21b262b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:28.068217 12593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da808a7f774e5ea7bd7ef298f1180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:28.071895 9544 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.071967 7797 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.072710 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.073472 7272 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.073762 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.074004 7667 consensus_queue.cc:237] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } }
I20240501 14:03:28.075055 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.081283 9813 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.082052 12132 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.085925 7664 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.086836 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.087276 7535 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.088449 9811 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.089781 7273 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.090150 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.090785 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.092294 12470 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.094847 9543 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.095999 12358 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.096832 12358 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.102227 9811 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.107520 7795 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.108532 12323 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.109086 9543 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.113298 12323 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.117265 12636 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.130103 7535 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.130638 7273 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.140892 7795 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.138504 12593 tablet_copy_client.cc:783] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:28.141562 12593 tablet_copy_client.cc:647] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:28.122778 7000 catalog_manager.cc:5484] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New cstate: current_term: 1 leader_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.142599 12398 ts_tablet_manager.cc:902] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:28.143038 12358 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.147737 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:28.150667 9676 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.151258 7137 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.151827 7405 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEARNER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.152093 12132 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.152361 12333 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.152943 12589 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.162938 7907 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e104d44c5fa4a29be94a33cf7e93ba6 with delete_type TABLET_DATA_TOMBSTONED (TS cc7e7386a9e540eabc4648e147073f1d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:50604
I20240501 14:03:28.145220 7795 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.166618 7140 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.166620 7535 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.166610 7929 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.167304 12589 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.167964 9543 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.168540 12579 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.170450 12579 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.170746 12589 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.174607 7139 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.184468 9794 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab9f466bd0244debff3e228ba99eb51 with delete_type TABLET_DATA_TOMBSTONED (TS 1c61c125cd144590ae1e54a2f87d9187 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48630
I20240501 14:03:28.187911 12668 ts_tablet_manager.cc:902] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
W20240501 14:03:28.193049 7865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.201:33735, user_credentials={real_user=slave}} blocked reactor thread for 73539.5us
I20240501 14:03:28.194689 7534 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.194861 9676 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.202289 6993 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.207226 7798 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.211694 12582 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.213351 7000 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:28.217222 12674 tablet_replica.cc:330] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: stopping tablet replica
I20240501 14:03:28.217813 12674 raft_consensus.cc:2229] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 14:03:28.218096 7862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.202:37399, user_credentials={real_user=slave}} blocked reactor thread for 96819.6us
I20240501 14:03:28.218377 12674 raft_consensus.cc:2258] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.219347 7798 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.220635 12674 ts_tablet_manager.cc:1875] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.225536 9676 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.233794 7138 raft_consensus.cc:1261] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.237154 12358 consensus_queue.cc:1035] T 4e104d44c5fa4a29be94a33cf7e93ba6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.237768 12593 tablet_copy_client.cc:523] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:28.248428 12579 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.249303 7405 consensus_queue.cc:237] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:28.251122 12593 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Bootstrap starting.
I20240501 14:03:28.254736 7796 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.254734 7535 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.254860 7139 raft_consensus.cc:2932] T 4e104d44c5fa4a29be94a33cf7e93ba6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:28.255419 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:28.256927 7339 consensus_peers.cc:485] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 -> Peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Couldn't send request to peer 704fb2d37343406790224dc76b40f210. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:28.256726 7140 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.261620 7929 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.263569 6992 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.272498 12679 tablet_replica.cc:330] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: stopping tablet replica
I20240501 14:03:28.273079 12679 raft_consensus.cc:2229] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:28.273804 12679 raft_consensus.cc:2258] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.274498 12668 tablet_copy_client.cc:308] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:28.275270 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 60cf39e6be52426c93c1534754be7264 from peer 1c61c125cd144590ae1e54a2f87d9187 ({username='slave'} at 127.22.5.202:35611)
I20240501 14:03:28.275452 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 60cf39e6be52426c93c1534754be7264 from peer 1c61c125cd144590ae1e54a2f87d9187 at {username='slave'} at 127.22.5.202:35611: session id = 1c61c125cd144590ae1e54a2f87d9187-60cf39e6be52426c93c1534754be7264
I20240501 14:03:28.278435 7425 tablet_copy_source_session.cc:205] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:28.279811 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:28.280903 9544 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.280663 7401 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.282781 7794 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.284982 7272 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.290079 7793 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.294900 9678 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.298475 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.305825 7926 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.306407 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.307108 9814 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.307700 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.311535 12315 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.315645 12668 tablet_copy_client.cc:783] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:28.315898 12668 tablet_copy_client.cc:647] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:28.311599 9543 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.318284 12668 tablet_copy_client.cc:523] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:28.318113 9814 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.316946 7139 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.321777 7666 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.322868 12674 ts_tablet_manager.cc:1888] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:28.323065 12674 log.cc:1196] T 4e104d44c5fa4a29be94a33cf7e93ba6 P cc7e7386a9e540eabc4648e147073f1d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/wals/4e104d44c5fa4a29be94a33cf7e93ba6
I20240501 14:03:28.324185 7272 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.324203 6985 catalog_manager.cc:4830] TS cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): tablet 4e104d44c5fa4a29be94a33cf7e93ba6 (table rebalance_test_table_1 [id=4aaaa50fde5b48afbb6f5a10b8826563]) successfully deleted
I20240501 14:03:28.327706 12668 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap starting.
I20240501 14:03:28.337447 6993 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.338203 7272 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.338892 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.340582 7273 consensus_queue.cc:237] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } }
I20240501 14:03:28.348129 7535 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.348423 7928 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.349144 12589 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.349463 12582 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.351518 7665 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.351830 12666 raft_consensus.cc:1050] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d: attempting to promote NON_VOTER 0d05240051574799acbbe13e21b262b4 to VOTER
I20240501 14:03:28.352129 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.353355 12666 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } }
I20240501 14:03:28.354259 7401 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.354890 12579 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.355485 12447 raft_consensus.cc:1050] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13: attempting to promote NON_VOTER 1c61c125cd144590ae1e54a2f87d9187 to VOTER
I20240501 14:03:28.356101 7796 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.356581 12579 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.356655 12447 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:28.364120 7667 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.364986 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.368213 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.369933 12398 tablet_copy_client.cc:308] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:28.370254 7139 raft_consensus.cc:1261] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer ceb055f6a1a540dcb61e1e11a157cdb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:28.370922 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 60cf39e6be52426c93c1534754be7264 from peer 704fb2d37343406790224dc76b40f210 ({username='slave'} at 127.22.5.196:56249)
I20240501 14:03:28.371065 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 60cf39e6be52426c93c1534754be7264 from peer 704fb2d37343406790224dc76b40f210 at {username='slave'} at 127.22.5.196:56249: session id = 704fb2d37343406790224dc76b40f210-60cf39e6be52426c93c1534754be7264
I20240501 14:03:28.373174 9811 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.374476 7425 tablet_copy_source_session.cc:205] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:28.375238 12549 consensus_queue.cc:1035] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.375751 12398 ts_tablet_manager.cc:1875] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:28.375800 7793 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.382498 12398 ts_tablet_manager.cc:1888] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:28.382824 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cf39e6be52426c93c1534754be7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:28.392107 9677 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.392861 12579 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.394069 12398 tablet_copy_client.cc:783] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:28.394357 12398 tablet_copy_client.cc:647] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:28.396353 12398 tablet_copy_client.cc:523] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:28.396736 12582 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.401651 12398 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:28.402369 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 253a0abd26814fb29f9adede664c5988 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:28.404932 7535 raft_consensus.cc:1203] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:03:28.405728 7535 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.410570 6992 catalog_manager.cc:5484] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:28.410312 7929 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.412786 9676 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.423238 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 253a0abd26814fb29f9adede664c5988 with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59468
W20240501 14:03:28.425766 7862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.194:41451, user_credentials={real_user=slave}} blocked reactor thread for 61052.7us
I20240501 14:03:28.426493 12631 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.427428 12631 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.429574 7797 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.437956 9811 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.439559 12707 tablet_replica.cc:330] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:28.440013 12707 raft_consensus.cc:2229] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:28.440469 12707 raft_consensus.cc:2258] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.442545 12707 ts_tablet_manager.cc:1875] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.449054 7273 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.457176 9543 raft_consensus.cc:1261] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.458051 12582 consensus_queue.cc:1035] T 253a0abd26814fb29f9adede664c5988 P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.460249 9544 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.466818 12707 ts_tablet_manager.cc:1888] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:28.467029 12707 log.cc:1196] T 253a0abd26814fb29f9adede664c5988 P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/253a0abd26814fb29f9adede664c5988
I20240501 14:03:28.467867 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet 253a0abd26814fb29f9adede664c5988 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:28.469070 7401 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.477234 9811 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.477355 7928 consensus_queue.cc:237] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } }
I20240501 14:03:28.477208 9678 raft_consensus.cc:2932] T 253a0abd26814fb29f9adede664c5988 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.478771 7272 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.478214 12396 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.479722 7140 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 LEARNER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.480486 12667 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.480710 9677 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.480975 7139 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.481247 12667 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.482064 7535 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.482126 12667 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.482571 7664 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.482885 12667 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.484931 7927 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:28.488229 9675 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.488286 7928 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.488236 7138 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.489917 12667 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.489799 7798 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.493638 7405 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.495306 7000 catalog_manager.cc:5484] T 5efbd81b803841a288eed30cb91dad31 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New cstate: current_term: 1 leader_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.499997 12323 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.505939 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd9f7d11351b4f8d84021720488abf25 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:28.506686 7664 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.507333 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.507579 7535 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.508142 7139 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.510334 7271 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.512842 9811 raft_consensus.cc:2932] T 5efbd81b803841a288eed30cb91dad31 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:28.513999 6999 catalog_manager.cc:5484] T bd9f7d11351b4f8d84021720488abf25 P 704fb2d37343406790224dc76b40f210 reported cstate change: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.518888 12679 ts_tablet_manager.cc:1875] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.521148 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f7d11351b4f8d84021720488abf25 with delete_type TABLET_DATA_TOMBSTONED (TS c8f737f3f6994db297b47fb23dad1b13 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33072
I20240501 14:03:28.512154 9677 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.528424 9543 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.534667 12716 tablet_replica.cc:330] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: stopping tablet replica
I20240501 14:03:28.535956 7273 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:28.540724 7666 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.541916 7405 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.543766 9814 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.544466 9812 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.547286 7665 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
W20240501 14:03:28.551054 7731 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.5.195:41999, user_credentials={real_user=slave}} blocked reactor thread for 116135us
I20240501 14:03:28.562702 12716 raft_consensus.cc:2229] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:28.563076 12716 raft_consensus.cc:2258] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.564225 12716 ts_tablet_manager.cc:1875] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.565265 9544 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.566040 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.566354 7666 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.567116 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.568012 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.568804 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.569653 7793 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.569674 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.570353 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.570497 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.541096 7272 raft_consensus.cc:1261] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.571313 9678 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.542997 7270 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.572182 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.586014 12398 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:28.586429 12398 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Bootstrap complete.
I20240501 14:03:28.586750 12398 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.185s user 0.051s sys 0.014s
I20240501 14:03:28.586938 7797 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.587533 12398 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:28.588330 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.588390 12666 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.589165 12398 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: LEARNER
I20240501 14:03:28.589452 12398 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:28.591946 7139 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.592532 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.593672 12398 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:03:28.595432 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 704fb2d37343406790224dc76b40f210-60cf39e6be52426c93c1534754be7264 received from {username='slave'} at 127.22.5.196:56249
I20240501 14:03:28.595585 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 704fb2d37343406790224dc76b40f210-60cf39e6be52426c93c1534754be7264 on tablet 60cf39e6be52426c93c1534754be7264 with peer 704fb2d37343406790224dc76b40f210
I20240501 14:03:28.597608 7272 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.599483 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.601068 7666 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.601909 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.607272 9811 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.608954 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.619357 12294 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.622196 12679 ts_tablet_manager.cc:1888] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:03:28.624863 9676 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.625564 7140 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.631798 6999 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 reported cstate change: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.631848 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.637435 7664 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.643958 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:28.645037 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.646086 12636 consensus_queue.cc:1035] T bd9f7d11351b4f8d84021720488abf25 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.658576 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.659371 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.660071 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.661353 12294 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.640777 9814 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.664655 7271 raft_consensus.cc:2932] T bd9f7d11351b4f8d84021720488abf25 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: false } } }
I20240501 14:03:28.669308 12666 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.676352 7664 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.677719 9811 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.678426 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dc517698d5a4ae19525c96334ce7417 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:28.678947 9678 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.680434 6998 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:28.635568 9544 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.686729 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.703449 7272 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.704857 9811 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.708705 12679 log.cc:1196] T 1ab9f466bd0244debff3e228ba99eb51 P 1c61c125cd144590ae1e54a2f87d9187: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-9/wal/wals/1ab9f466bd0244debff3e228ba99eb51
I20240501 14:03:28.717885 7798 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.726892 6984 catalog_manager.cc:4830] TS 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202:37399): tablet 1ab9f466bd0244debff3e228ba99eb51 (table rebalance_test_table_4 [id=c14454e484104ccfb04827433fa39ccf]) successfully deleted
I20240501 14:03:28.726833 12323 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.732568 7140 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.736804 12716 ts_tablet_manager.cc:1888] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:28.737691 12398 ts_tablet_manager.cc:902] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Initiating tablet copy from peer ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999)
I20240501 14:03:28.739445 7272 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.739830 6997 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:28.748828 9814 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.751102 12716 log.cc:1196] T bd9f7d11351b4f8d84021720488abf25 P c8f737f3f6994db297b47fb23dad1b13: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-5/wal/wals/bd9f7d11351b4f8d84021720488abf25
I20240501 14:03:28.755300 7797 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.760160 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.761348 7665 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.774957 7271 consensus_queue.cc:237] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } }
I20240501 14:03:28.775434 7401 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.777480 7666 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1c61c125cd144590ae1e54a2f87d9187 (127.22.5.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:28.793151 7926 consensus_queue.cc:237] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } }
I20240501 14:03:28.794144 6984 catalog_manager.cc:4830] TS c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198:45423): tablet bd9f7d11351b4f8d84021720488abf25 (table rebalance_test_table_3 [id=0dad7aaa69ee45d2aadcfb78af50751d]) successfully deleted
I20240501 14:03:28.794350 9541 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0d05240051574799acbbe13e21b262b4 (127.22.5.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { replace: true } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.796675 7140 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.797199 12631 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.799793 7797 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.800851 12631 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.803825 7272 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.805364 9544 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.806347 12631 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.807917 12631 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.813776 12666 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.815513 7272 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.820605 7667 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.821398 12636 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.822734 9678 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.822944 7929 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.823665 12579 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.824343 12579 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.824658 7667 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.828982 7797 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.831058 12582 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.842378 7139 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.842947 7798 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.845441 7535 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.846016 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:28.846144 12582 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.850124 9811 raft_consensus.cc:1261] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:28.850534 6999 catalog_manager.cc:5484] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:03:28.852896 12631 consensus_queue.cc:1035] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:28.853399 9524 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc517698d5a4ae19525c96334ce7417 with delete_type TABLET_DATA_TOMBSTONED (TS cc3a47f051a546a5a7788d088f061dd5 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33966
I20240501 14:03:28.856374 9811 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.857772 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab7d2ddba3440f0a1311f9eda7d3930 with delete_type TABLET_DATA_TOMBSTONED (TS ceb055f6a1a540dcb61e1e11a157cdb8 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:51150
I20240501 14:03:28.861626 9544 raft_consensus.cc:2932] T 1ab7d2ddba3440f0a1311f9eda7d3930 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } attrs { promote: false } } }
I20240501 14:03:28.865959 12761 tablet_replica.cc:330] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: stopping tablet replica
I20240501 14:03:28.868692 12761 raft_consensus.cc:2229] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:28.870345 12761 raft_consensus.cc:2258] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.873569 12761 ts_tablet_manager.cc:1875] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.876343 7401 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.878099 12582 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 14:03:28.882118 7206 consensus_peers.cc:485] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc -> Peer db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197:42421): Couldn't send request to peer db6b830d9bbe4451bd349c6ad6e14bab. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:28.887935 9814 raft_consensus.cc:1261] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Refusing update from remote peer b3b866abc97f437e8f24eccb653b3dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:03:28.889303 12582 consensus_queue.cc:1035] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:28.903990 12604 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:28.904551 12604 tablet_bootstrap.cc:492] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Bootstrap complete.
I20240501 14:03:28.904860 12604 ts_tablet_manager.cc:1367] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Time spent bootstrapping tablet: real 1.030s user 0.068s sys 0.001s
I20240501 14:03:28.906314 12604 raft_consensus.cc:348] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } }
I20240501 14:03:28.906636 12604 raft_consensus.cc:729] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8f737f3f6994db297b47fb23dad1b13, State: Initialized, Role: LEARNER
I20240501 14:03:28.906869 12604 consensus_queue.cc:260] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: NON_VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: true } }
I20240501 14:03:28.915870 12761 ts_tablet_manager.cc:1888] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:28.918301 12604 ts_tablet_manager.cc:1398] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 14:03:28.919960 12760 tablet_replica.cc:330] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: stopping tablet replica
I20240501 14:03:28.920488 7687 tablet_copy_service.cc:342] P db6b830d9bbe4451bd349c6ad6e14bab: Request end of tablet copy session c8f737f3f6994db297b47fb23dad1b13-210980fcec0c4358943ddd2cc7e935a6 received from {username='slave'} at 127.22.5.198:60205
I20240501 14:03:28.920544 12760 raft_consensus.cc:2229] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:28.920650 7687 tablet_copy_service.cc:433] P db6b830d9bbe4451bd349c6ad6e14bab: ending tablet copy session c8f737f3f6994db297b47fb23dad1b13-210980fcec0c4358943ddd2cc7e935a6 on tablet 210980fcec0c4358943ddd2cc7e935a6 with peer c8f737f3f6994db297b47fb23dad1b13
I20240501 14:03:28.920557 12761 log.cc:1196] T 1ab7d2ddba3440f0a1311f9eda7d3930 P ceb055f6a1a540dcb61e1e11a157cdb8: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ab7d2ddba3440f0a1311f9eda7d3930
I20240501 14:03:28.921029 12760 raft_consensus.cc:2258] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:28.922777 12760 ts_tablet_manager.cc:1875] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:28.925210 6984 catalog_manager.cc:4830] TS ceb055f6a1a540dcb61e1e11a157cdb8 (127.22.5.195:41999): tablet 1ab7d2ddba3440f0a1311f9eda7d3930 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:28.941409 12579 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.948143 9678 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 41ab226fa6b7403b8b44642e9470517d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.950596 7401 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P ceb055f6a1a540dcb61e1e11a157cdb8 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.953394 6998 catalog_manager.cc:5484] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc reported cstate change: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New cstate: current_term: 2 leader_uuid: "b3b866abc97f437e8f24eccb653b3dcc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:28.955622 9812 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 1c61c125cd144590ae1e54a2f87d9187 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.966604 7798 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P c8f737f3f6994db297b47fb23dad1b13 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.962615 7926 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P cc7e7386a9e540eabc4648e147073f1d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.976454 7535 raft_consensus.cc:2932] T 2264386e494644aa84e8109c816aaefa P 704fb2d37343406790224dc76b40f210 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER db6b830d9bbe4451bd349c6ad6e14bab (127.22.5.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: NON_VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { promote: true } } }
I20240501 14:03:28.981998 12398 tablet_copy_client.cc:308] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.195:41999
I20240501 14:03:28.982650 7425 tablet_copy_service.cc:140] P ceb055f6a1a540dcb61e1e11a157cdb8: Received BeginTabletCopySession request for tablet 5efbd81b803841a288eed30cb91dad31 from peer 704fb2d37343406790224dc76b40f210 ({username='slave'} at 127.22.5.196:56249)
I20240501 14:03:28.982820 7425 tablet_copy_service.cc:161] P ceb055f6a1a540dcb61e1e11a157cdb8: Beginning new tablet copy session on tablet 5efbd81b803841a288eed30cb91dad31 from peer 704fb2d37343406790224dc76b40f210 at {username='slave'} at 127.22.5.196:56249: session id = 704fb2d37343406790224dc76b40f210-5efbd81b803841a288eed30cb91dad31
I20240501 14:03:28.985661 7425 tablet_copy_source_session.cc:205] T 5efbd81b803841a288eed30cb91dad31 P ceb055f6a1a540dcb61e1e11a157cdb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:28.986995 12398 ts_tablet_manager.cc:1875] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:28.993666 12398 ts_tablet_manager.cc:1888] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:28.994069 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efbd81b803841a288eed30cb91dad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:28.997341 12398 tablet_copy_client.cc:783] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:28.997783 12398 tablet_copy_client.cc:647] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:29.001292 12398 tablet_copy_client.cc:523] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:29.004846 12398 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Bootstrap starting.
I20240501 14:03:29.014703 12668 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:29.015182 12668 tablet_bootstrap.cc:492] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Bootstrap complete.
I20240501 14:03:29.015491 12668 ts_tablet_manager.cc:1367] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Time spent bootstrapping tablet: real 0.688s user 0.064s sys 0.002s
I20240501 14:03:29.016304 12668 raft_consensus.cc:348] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:29.016661 12668 raft_consensus.cc:729] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c61c125cd144590ae1e54a2f87d9187, State: Initialized, Role: LEARNER
I20240501 14:03:29.017695 12515 raft_consensus.cc:1050] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4: attempting to promote NON_VOTER 704fb2d37343406790224dc76b40f210 to VOTER
I20240501 14:03:29.018922 12515 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } }
I20240501 14:03:29.016914 12668 consensus_queue.cc:260] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:29.035265 12668 ts_tablet_manager.cc:1398] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240501 14:03:29.036826 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 1c61c125cd144590ae1e54a2f87d9187-60cf39e6be52426c93c1534754be7264 received from {username='slave'} at 127.22.5.202:35611
I20240501 14:03:29.036955 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 1c61c125cd144590ae1e54a2f87d9187-60cf39e6be52426c93c1534754be7264 on tablet 60cf39e6be52426c93c1534754be7264 with peer 1c61c125cd144590ae1e54a2f87d9187
I20240501 14:03:29.038697 12760 ts_tablet_manager.cc:1888] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:29.039145 12760 log.cc:1196] T 2dc517698d5a4ae19525c96334ce7417 P cc3a47f051a546a5a7788d088f061dd5: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-7/wal/wals/2dc517698d5a4ae19525c96334ce7417
I20240501 14:03:29.039196 7926 consensus_queue.cc:237] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } }
I20240501 14:03:29.040143 7535 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.040498 6984 catalog_manager.cc:4830] TS cc3a47f051a546a5a7788d088f061dd5 (127.22.5.200:43849): tablet 2dc517698d5a4ae19525c96334ce7417 (table rebalance_test_table_2 [id=9aed4fa8cf794c97a111cddd84abd89f]) successfully deleted
I20240501 14:03:29.041268 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.045434 7401 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.046119 12666 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.050232 9678 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.050843 12666 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.055298 9812 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.055301 7665 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.055958 12636 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.056303 7797 raft_consensus.cc:1203] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:29.056538 12636 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.053891 7535 raft_consensus.cc:1203] T 60cf39e6be52426c93c1534754be7264 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:03:29.059602 12323 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.064033 9812 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.066593 7665 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.069628 6985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5396b2071184070bcb14763e1df7b5f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:29.070454 7272 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.071120 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.071241 9678 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.071825 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.073804 7797 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.074311 7271 raft_consensus.cc:1261] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.075275 12631 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.075835 12667 consensus_queue.cc:1035] T c5396b2071184070bcb14763e1df7b5f P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.076313 7401 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.076886 9676 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.079644 12398 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:29.080081 12398 tablet_bootstrap.cc:492] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Bootstrap complete.
I20240501 14:03:29.080364 12398 ts_tablet_manager.cc:1367] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Time spent bootstrapping tablet: real 0.076s user 0.045s sys 0.011s
I20240501 14:03:29.081200 12398 raft_consensus.cc:348] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:29.081498 12398 raft_consensus.cc:729] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 704fb2d37343406790224dc76b40f210, State: Initialized, Role: LEARNER
I20240501 14:03:29.081712 12398 consensus_queue.cc:260] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:29.083094 12398 ts_tablet_manager.cc:1398] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240501 14:03:29.083731 7425 tablet_copy_service.cc:342] P ceb055f6a1a540dcb61e1e11a157cdb8: Request end of tablet copy session 704fb2d37343406790224dc76b40f210-5efbd81b803841a288eed30cb91dad31 received from {username='slave'} at 127.22.5.196:56249
I20240501 14:03:29.083861 7425 tablet_copy_service.cc:433] P ceb055f6a1a540dcb61e1e11a157cdb8: ending tablet copy session 704fb2d37343406790224dc76b40f210-5efbd81b803841a288eed30cb91dad31 on tablet 5efbd81b803841a288eed30cb91dad31 with peer 704fb2d37343406790224dc76b40f210
I20240501 14:03:29.083649 7797 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.086529 7271 raft_consensus.cc:2932] T c5396b2071184070bcb14763e1df7b5f P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.092712 7798 consensus_queue.cc:237] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } }
I20240501 14:03:29.094576 6999 catalog_manager.cc:5484] T c5396b2071184070bcb14763e1df7b5f P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 12 to 13, VOTER 0d05240051574799acbbe13e21b262b4 (127.22.5.193) evicted. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } attrs { promote: false } } }
I20240501 14:03:29.097618 7928 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.097640 7929 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.097837 7139 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.098393 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.099706 12718 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.102015 12718 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.106420 9544 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.106979 7667 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.107692 9812 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.111994 12717 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.112406 12708 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.112828 12708 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.115307 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet c5396b2071184070bcb14763e1df7b5f with delete_type TABLET_DATA_TOMBSTONED (TS 0d05240051574799acbbe13e21b262b4 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59468
I20240501 14:03:29.120873 7271 raft_consensus.cc:1261] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer c8f737f3f6994db297b47fb23dad1b13: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.121515 12718 consensus_queue.cc:1035] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.125296 12718 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.133008 7665 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.133050 9544 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc3a47f051a546a5a7788d088f061dd5 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.136389 9812 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.149858 7271 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.150333 7000 catalog_manager.cc:5484] T 64b9b1b64def483aaed2a03f7d356915 P 1c61c125cd144590ae1e54a2f87d9187 reported cstate change: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New cstate: current_term: 1 leader_uuid: "c8f737f3f6994db297b47fb23dad1b13" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.153663 6984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64b9b1b64def483aaed2a03f7d356915 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:29.155292 7139 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.175063 7797 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.178027 7929 raft_consensus.cc:2932] T 64b9b1b64def483aaed2a03f7d356915 P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 41ab226fa6b7403b8b44642e9470517d (127.22.5.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } }
I20240501 14:03:29.182981 7665 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.178040 9814 raft_consensus.cc:1203] T 60cf39e6be52426c93c1534754be7264 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:03:29.175974 9812 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:03:29.190829 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.191375 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.194667 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:03:29.196668 12593 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:29.197196 12593 tablet_bootstrap.cc:492] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Bootstrap complete.
I20240501 14:03:29.199893 12593 ts_tablet_manager.cc:1367] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Time spent bootstrapping tablet: real 0.949s user 0.048s sys 0.014s
I20240501 14:03:29.201489 12593 raft_consensus.cc:348] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:29.201918 12593 raft_consensus.cc:729] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc3a47f051a546a5a7788d088f061dd5, State: Initialized, Role: LEARNER
I20240501 14:03:29.202165 12593 consensus_queue.cc:260] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false replace: true } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: NON_VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: true } }
I20240501 14:03:29.202975 12593 ts_tablet_manager.cc:1398] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:03:29.203677 7161 tablet_copy_service.cc:342] P 0d05240051574799acbbe13e21b262b4: Request end of tablet copy session cc3a47f051a546a5a7788d088f061dd5-11da808a7f774e5ea7bd7ef298f1180e received from {username='slave'} at 127.22.5.200:38709
I20240501 14:03:29.203814 7161 tablet_copy_service.cc:433] P 0d05240051574799acbbe13e21b262b4: ending tablet copy session cc3a47f051a546a5a7788d088f061dd5-11da808a7f774e5ea7bd7ef298f1180e on tablet 11da808a7f774e5ea7bd7ef298f1180e with peer cc3a47f051a546a5a7788d088f061dd5
I20240501 14:03:29.206970 12788 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.214509 12787 tablet_replica.cc:330] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: stopping tablet replica
I20240501 14:03:29.214745 12787 raft_consensus.cc:2229] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:29.215013 12787 raft_consensus.cc:2258] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:29.214821 9678 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.216024 12787 ts_tablet_manager.cc:1875] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:29.215842 7534 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.219662 7929 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.228719 7665 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.239240 6995 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab reported cstate change: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.246683 12804 ts_tablet_manager.cc:902] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Initiating tablet copy from peer b3b866abc97f437e8f24eccb653b3dcc (127.22.5.194:41451)
I20240501 14:03:29.246651 9544 raft_consensus.cc:1203] T 11da808a7f774e5ea7bd7ef298f1180e P cc3a47f051a546a5a7788d088f061dd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:29.252542 12787 ts_tablet_manager.cc:1888] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:29.259845 9658 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b9b1b64def483aaed2a03f7d356915 with delete_type TABLET_DATA_TOMBSTONED (TS 41ab226fa6b7403b8b44642e9470517d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:46848
I20240501 14:03:29.262629 7272 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.269986 7535 raft_consensus.cc:1203] T 5efbd81b803841a288eed30cb91dad31 P 704fb2d37343406790224dc76b40f210 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:03:29.282689 12807 tablet_replica.cc:330] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: stopping tablet replica
I20240501 14:03:29.283349 12807 raft_consensus.cc:2229] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:29.293871 12807 raft_consensus.cc:2258] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:29.296866 12787 log.cc:1196] T c5396b2071184070bcb14763e1df7b5f P 0d05240051574799acbbe13e21b262b4: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5396b2071184070bcb14763e1df7b5f
I20240501 14:03:29.299734 7929 consensus_queue.cc:237] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } }
I20240501 14:03:29.302022 7797 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.304845 7271 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240501 14:03:29.305737 7863 consensus_peers.cc:485] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d -> Peer 704fb2d37343406790224dc76b40f210 (127.22.5.196:37547): Couldn't send request to peer 704fb2d37343406790224dc76b40f210. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:29.305801 7401 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:03:29.306918 7798 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:03:29.307319 7665 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:03:29.307395 9812 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:03:29.308036 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.308113 9678 raft_consensus.cc:1261] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer cc7e7386a9e540eabc4648e147073f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:03:29.308617 12323 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.309923 6986 catalog_manager.cc:4830] TS 0d05240051574799acbbe13e21b262b4 (127.22.5.193:41299): tablet c5396b2071184070bcb14763e1df7b5f (table rebalance_test_table_6 [id=89d9f6f896bc4681b7665f97a5f67575]) successfully deleted
I20240501 14:03:29.310465 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.310986 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.311462 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.311868 12636 consensus_queue.cc:1035] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:03:29.312695 12807 ts_tablet_manager.cc:1875] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:29.318006 9812 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 704fb2d37343406790224dc76b40f210 (127.22.5.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.321264 12666 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.325325 7139 consensus_queue.cc:237] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } }
I20240501 14:03:29.326588 7798 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.327764 9812 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.330328 7667 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.332260 9814 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.332897 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.333684 9676 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.336223 7534 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.336796 12622 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.339902 6992 catalog_manager.cc:5484] T 2dc517698d5a4ae19525c96334ce7417 P cc7e7386a9e540eabc4648e147073f1d reported cstate change: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New cstate: current_term: 1 leader_uuid: "cc7e7386a9e540eabc4648e147073f1d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:29.342700 7271 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.343269 12788 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.344679 9678 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.345446 7401 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.345721 12622 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.346678 12515 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.348359 9677 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.354557 7793 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.355439 12515 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.358091 7793 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P c8f737f3f6994db297b47fb23dad1b13 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.360083 6986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b227b60efb7343bba8c4e89f81c3f6ff with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:03:29.363852 7271 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.364050 6998 catalog_manager.cc:5484] T b227b60efb7343bba8c4e89f81c3f6ff P 41ab226fa6b7403b8b44642e9470517d reported cstate change: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New cstate: current_term: 1 leader_uuid: "0d05240051574799acbbe13e21b262b4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.372383 7907 tablet_service.cc:1514] Processing DeleteTablet for tablet b227b60efb7343bba8c4e89f81c3f6ff with delete_type TABLET_DATA_TOMBSTONED (TS cc7e7386a9e540eabc4648e147073f1d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:50604
I20240501 14:03:29.384981 7271 raft_consensus.cc:2932] T 2dc517698d5a4ae19525c96334ce7417 P b3b866abc97f437e8f24eccb653b3dcc [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 704fb2d37343406790224dc76b40f210 (127.22.5.196) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: NON_VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: true } } }
I20240501 14:03:29.399438 12811 tablet_replica.cc:330] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: stopping tablet replica
I20240501 14:03:29.400475 12811 raft_consensus.cc:2229] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:29.401034 12811 raft_consensus.cc:2258] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:29.409148 7667 raft_consensus.cc:1261] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Refusing update from remote peer 0d05240051574799acbbe13e21b262b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:03:29.409624 12811 ts_tablet_manager.cc:1875] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:29.409868 12515 consensus_queue.cc:1035] T b227b60efb7343bba8c4e89f81c3f6ff P 0d05240051574799acbbe13e21b262b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:03:29.411620 7665 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P db6b830d9bbe4451bd349c6ad6e14bab [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.414057 7534 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 704fb2d37343406790224dc76b40f210 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.427441 12804 tablet_copy_client.cc:308] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Beginning tablet copy session from remote peer at address 127.22.5.194:41451
I20240501 14:03:29.428692 7293 tablet_copy_service.cc:140] P b3b866abc97f437e8f24eccb653b3dcc: Received BeginTabletCopySession request for tablet 2264386e494644aa84e8109c816aaefa from peer db6b830d9bbe4451bd349c6ad6e14bab ({username='slave'} at 127.22.5.197:48613)
I20240501 14:03:29.428915 7293 tablet_copy_service.cc:161] P b3b866abc97f437e8f24eccb653b3dcc: Beginning new tablet copy session on tablet 2264386e494644aa84e8109c816aaefa from peer db6b830d9bbe4451bd349c6ad6e14bab at {username='slave'} at 127.22.5.197:48613: session id = db6b830d9bbe4451bd349c6ad6e14bab-2264386e494644aa84e8109c816aaefa
I20240501 14:03:29.432363 9812 raft_consensus.cc:2932] T b227b60efb7343bba8c4e89f81c3f6ff P 1c61c125cd144590ae1e54a2f87d9187 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER cc7e7386a9e540eabc4648e147073f1d (127.22.5.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: false } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } } }
I20240501 14:03:29.434695 7293 tablet_copy_source_session.cc:205] T 2264386e494644aa84e8109c816aaefa P b3b866abc97f437e8f24eccb653b3dcc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:29.436220 12807 ts_tablet_manager.cc:1888] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:29.436415 12807 log.cc:1196] T 64b9b1b64def483aaed2a03f7d356915 P 41ab226fa6b7403b8b44642e9470517d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-8/wal/wals/64b9b1b64def483aaed2a03f7d356915
I20240501 14:03:29.437435 12804 ts_tablet_manager.cc:1875] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:29.453281 6987 catalog_manager.cc:4830] TS 41ab226fa6b7403b8b44642e9470517d (127.22.5.201:33735): tablet 64b9b1b64def483aaed2a03f7d356915 (table rebalance_test_table_0 [id=2e10dcfc24d546809f1b797c1176e79a]) successfully deleted
I20240501 14:03:29.497879 12804 ts_tablet_manager.cc:1888] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:03:29.498483 12804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2264386e494644aa84e8109c816aaefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:29.480685 12811 ts_tablet_manager.cc:1888] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 14:03:29.508423 12811 log.cc:1196] T b227b60efb7343bba8c4e89f81c3f6ff P cc7e7386a9e540eabc4648e147073f1d: Deleting WAL directory at /tmp/dist-test-taskA1ASV2/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714571955458352-22551-0/raft_consensus-itest-cluster/ts-6/wal/wals/b227b60efb7343bba8c4e89f81c3f6ff
I20240501 14:03:29.514430 6985 catalog_manager.cc:4830] TS cc7e7386a9e540eabc4648e147073f1d (127.22.5.199:33993): tablet b227b60efb7343bba8c4e89f81c3f6ff (table rebalance_test_table_5 [id=251b249c1e9e48c7866300402b706715]) successfully deleted
I20240501 14:03:29.520475 12804 tablet_copy_client.cc:783] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:29.520845 12804 tablet_copy_client.cc:647] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:29.533213 12804 tablet_copy_client.cc:523] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:29.551354 12804 tablet_bootstrap.cc:492] T 2264386e494644aa84e8109c816aaefa P db6b830d9bbe4451bd349c6ad6e14bab: Bootstrap starting.
I20240501 14:03:29.571285 12470 raft_consensus.cc:1050] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab: attempting to promote NON_VOTER c8f737f3f6994db297b47fb23dad1b13 to VOTER
I20240501 14:03:29.572559 12470 consensus_queue.cc:237] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: false } }
I20240501 14:03:29.599154 7793 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P c8f737f3f6994db297b47fb23dad1b13 [term 1 LEARNER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:29.600097 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:29.604213 9676 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 41ab226fa6b7403b8b44642e9470517d [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:29.604878 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:29.613442 12549 raft_consensus.cc:1050] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8: attempting to promote NON_VOTER 704fb2d37343406790224dc76b40f210 to VOTER
I20240501 14:03:29.614763 12549 consensus_queue.cc:237] T 60cf39e6be52426c93c1534754be7264 P ceb055f6a1a540dcb61e1e11a157cdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } attrs { replace: true } } peers { permanent_uuid: "c8f737f3f6994db297b47fb23dad1b13" member_type: VOTER last_known_addr { host: "127.22.5.198" port: 45423 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } } peers { permanent_uuid: "b3b866abc97f437e8f24eccb653b3dcc" member_type: VOTER last_known_addr { host: "127.22.5.194" port: 41451 } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "cc3a47f051a546a5a7788d088f061dd5" member_type: VOTER last_known_addr { host: "127.22.5.200" port: 43849 } attrs { promote: false } } peers { permanent_uuid: "41ab226fa6b7403b8b44642e9470517d" member_type: VOTER last_known_addr { host: "127.22.5.201" port: 33735 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c61c125cd144590ae1e54a2f87d9187" member_type: NON_VOTER last_known_addr { host: "127.22.5.202" port: 37399 } attrs { promote: true } } peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } attrs { promote: false } }
I20240501 14:03:29.619865 7139 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P 0d05240051574799acbbe13e21b262b4 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:29.621492 7401 raft_consensus.cc:1261] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Refusing update from remote peer db6b830d9bbe4451bd349c6ad6e14bab: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:29.622689 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:29.623814 12358 consensus_queue.cc:1035] T 210980fcec0c4358943ddd2cc7e935a6 P db6b830d9bbe4451bd349c6ad6e14bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:29.676067 7405 raft_consensus.cc:2932] T 210980fcec0c4358943ddd2cc7e935a6 P ceb055f6a1a540dcb61e1e11a157cdb8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c8f737f3f6994db297b47fb23dad1b13 (127.22.5.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "704fb2d37343406790224dc76b40f210" member_type: VOTER last_known_addr { host: "127.22.5.196" port: 37547 } } peers { permanent_uuid: "db6b830d9bbe4451bd349c6ad6e14bab" member_type: VOTER last_known_addr { host: "127.22.5.197" port: 42421 } } peers { permanent_uuid: "cc7e7386a9e540eabc4648e147073f1d" member_type: VOTER last_known_addr { host: "127.22.5.199" port: 33993 } attrs { replace: true } } peers { permanent_uuid: "ceb055f6a1a540dcb61e1e11a157cdb8" member_type: VOTER last_known_addr { host: "127.22.5.195" port: 41999 } } peers { permanent_uuid: "0d05240051574799acbbe13e21b262b4" member_type: VOTER last_known_addr { host: "127.22.5.193"